Hey there friendly blog reader, we’re taking a survey!
- How many times did you look in the mirror last year?
- How many times did you go to the gym last week?
Go ahead and think about it…we’ll wait…
Give up? Here’s the problem with surveys that want to measure past behavior. Surveys depend on respondents telling you the truth about what they did.
So why wouldn’t they tell you the truth?
Reason #1: The respondent has no idea what the truth is.
Think about our first survey question–“How many times did you look in the mirror last year?”
This is a really tough question because, frankly, most people have no idea how many times they’ve looked in the mirror over the course of a year. This means they’ll end up wildly guessing and your data will be just as meaningless as a measure of actual mirror glances.
How do we fix it? Here are some tricks:
a. In general, providing a specific, bounded timeframe as context will help your respondents answer your questions accurately. A good litmus test is trying to answer the question yourself–chances are if you don’t know how many times you looked in the mirror last year, your respondents won’t know either! Give respondents a manageable timeframe. Estimating how many times you look in a mirror per day is a lot easier to wrap your head around.
If what you, as the researcher, are interested in is yearly behavior patterns, simply multiply the estimates you get by 365 and you have your yearly estimates! To focus your respondents even more on the right timeframe, ask them about the timeframe immediately preceding the one you’re interested in and then the one you actually care about. (For example, “How many times did you look in the mirror yesterday?” followed by “How many times did you look in the mirror today?”)
b. Sometimes we have a vague sense of what the number might be, but not an exact one. To get around the difficulty of recalling a specific number of times something occurred or a specific amount of time it occurred, you can use a set of response options that have ranges (2-4 hours or 1-5 times or 3-10 days, etc.) that allow for the fuzziness of memory. Or you can use a Likert scale with word labels. This will give you a measure of how often your respondent did a certain behavior.
Keep in mind, however, that “often” can be a very subjective judgment. For some people, looking in the mirror ten times a day is extremely often, for others not so much. Without numbers you are now measuring the perceptions that your respondents have of their own behaviors. This can either be a blessing or a headache depending on your research question.
Reason #2: The truth is unflattering for the respondent.
Think about our second survey question–“How many times did you go to the gym last week?”
People might not answer this question accurately because they’re worried their answer will make them look bad. This discomfort with admitting the truth, even to an anonymous survey is very common, and psychologists call it social desirability bias. When it comes to unsavory or unflattering past behaviors—not eating healthy, cheating on a test, lying to a friend, etc.—respondents may feel uncomfortable telling the truth.
Thankfully, for all you survey makers, there are two different solutions to this problem. We covered this in our blog post on sensitive topics, but here’s a quick recap.
How do we fix it? Here are some tricks:
a. Remind the respondent the survey is anonymous (or at least confidential if you’re collecting identifying information. This will lessen the feeling for the respondent that someone’s judging his or her answers.
b. Normalize the behavior with some introductory text that takes away the social stigma from whatever behavior you’re asking about (For example, “In talking to people about cheating on tests, we’ve found that people often cheated on tests in school because they were too busy or tired to study. How about you, have you ever cheated on a test?”).
c. Allow the respondent to express how much he or she plans to do something in the future. (For example, “How many times did you go to the gym last week? followed by, “How many times did you plan to go to the gym next week?”) This allows the respondent to present a socially desirable side, alleviating the need to gloss over having behaved differently in the previous time interval.
Try out our solutions for reporting behaviors in your next survey, and don’t forget to let us know how it went!
