Putting our Audience survey panel quality to the test
Here’s how our research team ensures your surveys receive thoughtful, consistent responses.
How we test for survey panel quality
From December 2015 through February 2016, we sent the same monthly survey to respondents in our SurveyMonkey Audience panel. Each survey ran for 1 week, included 25 questions, and received more than 1,000 responses.
We compared the results for each wave, weighting for age and gender, to see if we got meaningful data back from our respondents. And we built our survey in order to test for 3 respondent behaviors that may negatively impact data quality:
- Straight lining: Behavior in which respondents select the same answer over and over again in a set of matrix questions
- Open-end response validity: Whether respondents write invalid responses that are gibberish, contain foul language, or give non-helpful comments
- Trap question: Completion of a picture-identification task, ensuring the respondent is paying attention
In addition to monitoring data quality, we wanted to make sure results were consistent from one wave to the next. Why? Even if members cycle in and out of Audience, we always want to be sure we maintain a panel that’s reliably representative of the general population.
Some of the questions we asked were included specifically because their responses should not change much over time, such as questions around:
- Physical characteristics: Different physical characteristics (e.g., green eyes, red hair, right-handedness)
- Purchasing and financial behaviors: Whether respondents own and use debit and credit cards, and how often they shop for certain items
- Opinions: What respondents think about the morality of alcohol use and homosexuality
- Demographics
We also checked to make sure the amount of time respondents took to complete the survey did not change significantly from wave to wave.
Data quality
As we mentioned before, in order to determine whether or not our panel yields quality data, we measured 3 key metrics: straight lining, open-ended responses, and a trap (picture-identification) question. Here’s what we found:
- 97% didn’t straightline responses
- 97% gave valid open-ended responses
- 94% passed the trap question
As you can see, 97% of the respondents in the U.S. passed our straight lining test. This means almost all respondents carefully differentiate the questions in a matrix (grid) question type when providing answers. Similarly, about only 2–3% of the open-ended responses are invalid or gibberish, while the majority of respondents provided a valid and meaningful answer.
We also included a trap question in the survey to test whether respondents paid close attention to the question—and found 94% of respondents passed this quality check as well. The question was asked more than halfway through the survey, suggesting that most respondents still paid enough attention to instructions and picked the correct answers when they got close to the end of the survey.
Response reliability
Next we examined whether the 3 surveys over the course of 3 months provide consistent findings. We tested for this by asking questions that should yield stable answers in terms of proportionality. For example, a sample should always include roughly the same percentage of people who say they have green eyes.
In the table below, we’ve included responses to these questions. For one question, respondents could “select all that apply,” so we show each of those response options separately. A check mark indicates no significant change over the 3 waves.
Most of the questions show a consistent pattern over time, suggesting the data collected from Audience are reliable and trustworthy. From 1 wave to another, of the 23 indicators we tracked, 20 did not change significantly.
| Survey question | Consistency |
| I have green eyes | 3 |
| I have naturally red hair | 3 |
| My birthday is in February | 3 |
| I have a twin | 3 |
| I am right-handed | 3 |
| I am less than 6 feet tall | 3 |
| I wear corrective lenses (glasses or contacts) | 3 |
| None of the above | å |
| Own a credit card | 3 |
| Own a debit card | 3 |
| Used a credit card in the past year | 3 |
| Used a debit card in the past year | 3 |
| Moral acceptability of alcohol use | å |
| Moral acceptability of homosexuality | 3 |
| Picture verification question | 3 |
| Education level | 3 |
| Race/ethnicity | 3 |
| Income | 3 |
| Motivation for taking the survey | 3 |
| Rating of survey quality | 3 |
| Open-ended question | 3 |
| Time to complete | å |
| Device completed on | 3 |
What this means for you
Quality: The 3 data quality indicators we examined all indicate SurveyMonkey Audience respondents provide thoughtful, high-quality responses. Only very small percentages of respondents failed our data quality checks.
Reliability: Over the 3 survey waves we ran, our results did not change substantially, indicating SurveyMonkey Audience provides consistent survey results representative of your target demographic regardless of when you’re conducting your survey.
This means you can survey away with confidence knowing you’ll get results you can trust! And remember: Follow Audience’s survey design guidelines to increase the likelihood of receiving thoughtful answers.
Survey your target market today
Get reliable feedback from qualified people in minutes.
