Why filtering your data by demographic groups is a game-changer

You’ve designed your survey, created it, collected data…now what? Well, depending on your needs, you could be fine just looking at overall results without filtering them at all.

But you might not have the whole picture. Practically anyone can get value out of looking closer at their survey data, and respondents’ demographic data is often the perfect place to start.

When you capture demographic data about your survey takers, you can build a profile of your respondents. It helps to understand how different groups of people answer your survey differently—whether they are members of your organization, students in a class, or employees at your company.

SurveyMonkey’s Analyze tool is perfect for uncovering the important differences between the demographic groups taking your survey (e.g. race/ethnicity, gender). Often these differences are the most interesting parts of your data and can make a big impact on the decisions your business or organization makes.

As survey researchers, we often look at a variety of subgroups in the analysis process. For instance, the Survey Research team at SurveyMonkey always asks what political party our respondents most identify with because we expect to see differences in responses by partisanship. Some evidence of when these differences mattered pop up when we study Donald Trump’s job approval ratings, who trusts the media, and activism behaviors.

At the 2017 AAPOR Annual Conference this year, researchers shared examples of how subgroup analyses changed the story entirely. Take a look at these two examples:

1. Race: University of Michigan researchers did several experiments looking at how different racial groups process the same information. They took a highly-charged and widely publicized issue—officer-involved shootings—and asked respondents questions about what they thought happened. They found that white and black respondents gave very different answers even when presented with the same information. If they had only looked at overall responses, these differences would be completely masked.

2. Religion: Experts at Pew Research Center looked at subgroups within a religion, specifically, people who identify as evangelical or “born again” Christian. While respondents were nearly identical on things like how often they pray and religious service attendance, black and white respondents gave polar opposite opinions on other issues. When it came to questions about party affiliation, white respondents tended to be more Republican, while black respondents were more often Democrat. It also impacted who they intended to vote for in 2016 (White respondents mostly favored Trump, Black respondents almost all favored Clinton). Finally, their views on social issues, such as same-sex marriage and abortion were significantly different: White respondents skewed more conservative across the board.

Both sets of research findings point us to the same conclusion: When you get your data, keep digging deeper and you may find another story.

Here are a few things YOU can do to get started with using SurveyMonkey Analyze.

Knowing what to ask in your surveys is key and we’re always looking to give you the tools you need to tell the whole story with your data. This means thinking about how you plan to analyze your data before you’ve even started building your survey. How do you do that? Just make sure you complete these three important steps before you deploy your survey:

Check that…

All necessary questions are present and accounted for.

Go back and make sure the questions you are interested in are included in your survey. This may seem obvious, but sometimes when you are working through different iterations of your survey with a lot of different people involved, an important question might get cut accidentally. To be sure that you can look at your data from a variety of angles, build into your survey-writing process a step where you check that all necessary demographic questions likely to have an impact on your data are included.

Your sample size is sufficient.

Do you have enough people from the population of interest? Let’s say you had 200 men and only 30 women reply to your survey. Don’t make claims about gender differences and business decisions based on a sample size so small that the findings cannot be generalizable. Use the SurveyMonkey Sample Size Calculator and Statistical Significance feature to help you along the way.

Then, compare but beware…

Differences by variable(s).  

Now that you you’ve deployed your survey and gotten all your responses back, you can use Analyze to compare between and within groups. For instance, if you asked respondents their gender, you now have a variable by which you can see how responses might differ (or not) by each group. Be careful not to make inferences with small numbers. You want to be sure that your results hold true and that you have a large enough sample to be able to say that differences are not simply due to chance.

And that’s it! Every time you send a survey, it’s worthwhile to take a closer  look at who answered which questions. That means using filters and crosstabs in Analyze to cut your data so you can see it from a new angle. And of course, the best place to start is often with demographic variables. Try filtering by demographics next time you create a survey!

For more interesting insights from the 2017 AAPOR conference, check out survey expert, Laura Wronski’s, article on how and when to ask about sexual orientation and gender identity in your surveys.