One of the first questions I wonder when I’m looking at survey results is: who was the audience? The second question: how large was the sample surveyed? Both of these are critical when looking at survey data and making inferences based on that data. You might have the best survey in the world, but if you didn’t get enough responses, you can’t generalize the answers to your audience.
We’ve featured a couple of insights into sample sizes in the past. First, we have a post about obtaining a “nationally representative” sample. In the post, the author talks about using quotas to help set up a nationally representative sample, but calls caution to the difference between how you set your quotas and how you analyze the results (hint: they should match exactly for best results). In another post, a different author goes more in-depth into sample sizes and makes the assertion that more is better, especially if you’re wanting to look at behavioral metrics.
Here, I’m taking a step back and looking at basic sample sizes. These are mathematical sample sizes, and, really, are best for obtaining general data. Let’s start with defining your audience.
LEARN ABOUT: Survey Sample Sizes
Getting your audience right first is critical in market research. For example, you have a questionnaire that is all about preschools in your area, and which preschools are most popular, best rated, least expensive. You would not want to ask people without preschool-aged children about the topic. Nor would you want to ask people whose youngest children are in college. Instead, your audience should include parents and caregivers of children ages 3-6 (asking those whose children are slightly older than preschool-aged will get you answers from those who already put their children through preschool, which can be relevant in this case if they sent their children to the preschools you are asking about in your study).
To make sure that these individuals are the ones answering your survey, there are two things you can do. First, you can purchase respondents via a panel. The second thing is ask screening questions such as, “How old are your children?” and “Are your children currently enrolled in preschool?” and “How long ago did your children attend preschool?”
In addition to making sure you have the right audience for your study, you also need to track how many responses you’re receiving to be sure that you’re getting the right number to provide valid insights. For this, I love this handy, free online sample size calculator. I use this calculator any time I need to get a sample size. The site has two sections, one that lets you enter the confidence level you want (how sure you want to be about your data) to determine the sample size needed, and the other section helps calculate the confidence interval for your data, based on the sample size you intend to use and the population size from which you’re getting your sample (such as the group of parents and caregivers with preschool age children in your area).
The section I use most is the first section – sample size I need based on the population I want to study and the confidence level I would like for my research. What this tells me is that I need at least that many responses, not that I need to survey at least that many people. If you get the number of responses equal to the sample size, you can infer predictions about the entire population you’re studying. If you get fewer responses, you can describe your sample, but you can’t necessarily apply the information to the rest of your audience.
Stay tuned for next week, when we’ll look at best practices for how many surveys you need to distribute in order to assure you get the number of responses you need based on the distribution method you intend to use (paper, online, email, social media).
