Have you ever launched a survey? Good. But have you ever heard of the term Total Survey Error (TSE)? Well, no worries—this blog’s for you!
In the fierce debate about what makes up a high-quality survey, one approach in our field has gained significant traction—thinking comprehensively.
Being mindful and thinking about all the possible sources of errors that can lead to incorrect findings is key.
Last week, we presented work from the SurveyMonkey survey research team at the inaugural International Total Survey Error Conference in lovely Baltimore. Here’s a breakdown of 3 major types of TSE to think about the next time you launch a survey.
Error 1: Undercoverage error
Ideally, a survey should start with a sampling frame, or a list of all the people in the target population. It can be a list of telephone numbers, postal addresses, or emails. In reality, however, no such list exists for the general population.
Census data probably make up the most comprehensive list of residents living in the country, but the raw data are not released to the public for privacy and confidentiality reasons.
Creating this type of list is a very expensive process and as with many things in life–it’s not always perfect. When some people are missing from the list, the potential undercoverage error may occur. In particular, if the people on the list have different opinions from people not on the list, the survey will produce biased results.
For example, a web survey will not be able to reach people without internet access. If the survey’s goal is to find out the general public’s opinion about new technology, it is very likely to result in biased findings.
Error 2: Nonresponse error
Say you’ve decided to move forward and launch your survey with that not-perfect list. There’s no guarantee that everyone invited to take the survey will respond. In fact, the response rates of most surveys have been declining for the past decade or so.
If the respondents and nonrespondents have different opinions, attitudes, or behaviors, the survey estimates based on respondents will not accurately reflect the reality of the entire population. When this happens, we say there’s a nonresponse error. A few things can help improve the response rate.
For example, offering different incentives, using different survey invitation strategies, or simply sending a better-designed survey can go a long way in increasing your response rate as well as potential nonresponse bias.
Error 3: Measurement error
Don’t assume your survey is going to be perfect even if you fix the coverage and nonresponse errors. Measurement error is the degree to which a survey estimate differs from its “true” value. Measurement errors impose a threat to the validity of the survey results.
Measurement error is very common in surveys; it’s actually more common than the other two errors. It’s often the consequence of the way questionnaires are designed. Leading question wordings, missing response options and misleading question orders are all common problems.
How to deal with these errors
Given the prevalence of these errors and the threats they impose on survey quality, survey researchers have spent a ton of time and effort to study them and try to correct them. Some errors, such as measurement errors, can potentially be prevented at the survey design stage.
A scientifically designed questionnaire that follows the best practices of survey methodology can really help reduce measurement error.
Nonresponse errors happen during the data collection stage, so a carefully planned data collection scheme can improve the response rate. Undercoverage errors are harder to deal with, especially for a general population survey. However, in the last few years, a handful of commercial sample providers started to compile and sell sample frames in the U.S.
If your survey is directed at your existing customers, all you really need to do is to carefully compile and organize your customer database inside your CRM and contact them by email.
Any questions about Total Survey Error? Let Mingnan know in the Comments below!
