So you’ve got your survey hot off the presses and ready to launch. There’s just one problem…who do you send it to?
Past blog posts have talked about what a “sample” of a population is and how your sample can affect your results. But let’s back up a minute and talk about how you should be getting your sample in the first place. The best, and most ideal, way to sample is to randomly sample your respondents.
What is “random sampling”?
Random sampling is a way to sample in which everyone in the population has a chance of being chosen for the sample, and whoever’s picked is chosen completely at random. This is great because there’s no bias – some people aren’t more likely to be picked than others. Plus, anyone from your population could conceivably be picked for your sample, so you can be more confident your results match what your population really thinks.
The simplest way to understand random sampling is to think of someone pulling slips of paper at random out of a hat. Every slip of paper in the hat has an equal chance of being plucked out. So if every slip of paper has a name on it, every name has an equal chance of getting picked. That means that it is “random” which names get picked.
How could you get a “random sample” of survey respondents?
You could…
- Approach random people on a random selection of streets at random times.
- Call randomly generated phone numbers at random times of the day.
- Mail out letters to randomly selected addresses from randomly selected regions.
- Email an online survey to randomly generated email addresses.
But, wait, there are some problems with this…
Let’s go back to our hat example. Do all the names in the hat really have an equal chance of being picked? Well…have you ever picked a slip of paper out of a hat? No one takes the slip of paper right on the top! You stick your hand way in there and grab one from at least the middle. So the papers on the top might not really have a fair shot at getting picked.
There are similar problems with surveying…
- Approaching people randomly is not only a little creepy but also unlikely to actually be random. Especially because the person you have doing the actual approaching is going to bias who says yes to the survey and potentially even the answers they give. (Don’t believe me? Try it yourself. Call up Kanye West and ask him to hand out a survey on one street—and then have your grandma hand out a survey on another street. Watch what happens.)
- People who answer phone calls and letters are a very particular subset of people these days and these methods may not be able to estimate some types of behavior.
- Methods 1-3 can also be very time consuming and very expensive and you’ve got yourself in a bit of a pickle.
- Going the email route, chances are you’ll be sent directly to a SPAM folder and people will never see your email at all. Chances are you’ll also get lots of angry emails about spamming random strangers.
So what can you do to fix these problems?
In a perfect world, everyone in our population would have a chance of being picked in our sample—in practice, that is virtually impossible. With surveys, as with many other things in life, you’ll need to strike a balance between what is ideal and what is possible in the real world. We like to call it “random(ish) sampling.” Here are some suggestions we have for random(ish) sampling…
- If you have a concrete “sampling frame” (a list of the names of the people in your population), you can use a random number generator to select which people to pick to survey from your list. (Want help with this? See how to do this in Excel.)
- If you don’t have a concrete sampling frame, you could vary who is approaching strangers on the street to ask them to fill out the survey. Have Kanye and your grandma go survey collecting together—and have them bring some friends to help too. The biases that each of them bring to data collection are likely to balance out.
- Send your survey out more than one way. Mail it, call people, send it online, shout it from the rooftops! Okay the last one was a joke, but the diversity of recruiting methods can help get different types of people to respond.
- Post your online survey on different types of websites where different types of people might see it. People who read political blogs may be different people from those who read fashion blogs. This is a nice alternative to the mass email that is likely to hit everyone’s spam filter. You can cast a wide net to draw in a diverse crowd without clogging anyone’s inbox.
