The Hidden Benefits of Using Surveys

Researchers love interviews for many reasons. We know that direct engagement with participants allows in-depth probing, and the flexibility of conversations takes us down paths we hadn’t considered. We also know that conducting interviews well is a skill that must be learned and honed over time. It’s not as simple as just sitting down and having a conversation.

In my work at Dscout enabling product designers and managers to run their own user research, I’ve found folks gravitating strongly to moderated methods. When I ask them why they prefer interviews over surveys, they often express their fear of missing out.

They’re afraid they’ll lose something in a survey they would have had access to one-on-one. I try to challenge them on this point of view, encouraging them to explore how a survey might capture the information they seek in a more streamlined way.

Jump to:

• The benefits of unmoderated surveys
• Common fears and misconceptions around surveys
• Example survey
• Parting thoughts

The benefits of unmoderated surveys

Interviews are often the right method, but not always! I prefer surveys most of the time. They’re especially helpful for non-researchers who do research.

✔ Less oversight needed

A trained researcher can’t realistically accompany a non-researcher conducting interviews, otherwise they would probably just do it themselves!

✔ Streamlined analysis

If you’re a designer running your own study, you probably want your data relatively quickly so you can go back to designing. With interviews, you’re often left with heaps of squishy information to synthesize.

Having some scales to start from can make a world of difference when digging into your study data. Surveys can help you make sense of your data faster, especially if you design them with analysis in mind.

✔ More efficient and effective

Once you’ve designed your survey, data collection is pretty hands-off! Surveys also guarantee everyone answers the same questions, which protects the quality of your data.

✔ Easier to conduct than interviews

Well-designed surveys can be better than interviews. But they must be well-designed! Surveys are often designed poorly, but fortunately, the principles of good survey design can be taught. I especially like the idea of non-researchers who do research using surveys more often.

Much of what we do as user researchers feels like a blend of art and science. Surveys are more science and interviews are more art. I think it’s easier to teach someone the fundamentals of how to ask appropriate written questions than it is to teach them all the skills required to conduct good user interviews: body language, timing, withholding the right amount of information, probing just enough, pulling it back in gently when it goes off the rails, being just conversational enough, and synthesizing heaps of qualitative data without cherry picking.

Common fears and misconceptions around surveys

1. What if I have follow-up questions?

Build the “Tell me more about that” approach into your survey design! Open-ended follow-up questions allow participants to give more context to rating scales or rankings.

If you’re worried about missing out, use follow-up questions to your advantage. Using a mix of closed and open-ended questions allows respondents to convey their thoughts using their own words. The order of your questions makes a difference, too. To avoid priming respondents, and think about the funnel method when you’re designing surveys: start broad and get more specific as you go.

2. Aren’t surveys too impersonal?

This can be a good thing! User researchers are often People Nerds who enjoy connecting with others, but an interview can be a big ask, especially for introverts. When not faced directly with the person consuming their feedback, some people might open up more in a survey than in a face-to-face scenario.

Make it as easy as possible to respond. Use plain language so everyone understands what you’re asking. Don’t overwhelm respondents with a million open-ended questions. Be strategic and concise, ask one question at a time, and leverage single-select rating scales.

3. Written responses aren’t as natural as spoken ones

You might worry that responses will be too considered in a survey, instead preferring the spontaneous nature of interviews. Surveys give respondents space to think and reflect, which may be a pro or a con depending on your goals.

The great thing about surveys is that you can do both. You can design questions that ask for their gut reactions without putting too much pressure or thought into their answers, in addition to questions that allow them to respond thoughtfully.

If you want to get quick first impressions, ask for them! One useful approach for this is using a short open-end. You can ask participants to describe first impressions in three words, then you can ask a long open-ended follow-up to allow them to expand more on why those words came to mind.

4. I’m seeking qualitative data, not quantitative

Qualitative and quantitative are not mutually exclusive concepts. Your survey can gather hard numbers and rich contextual data if you're creative and thoughtful.

Blend methods! Use surveys as a baseline and dig deeper with interviews if you want. With Diary you can ask follow-up questions, or you can always invite folks to a live mission after. Use a mixture of closed and open-ended questions. On Dscout, you can also use media questions (photos, videos, or screen recordings) to see and hear feedback from the participant’s point of view.

5. What if I leave something out?

Have someone review your survey before you send it out into the world. You can have a researcher review your interview guide, but there is potential for extreme variation in your planned conversation and how it actually goes.

With surveys, you can send out exactly what you want with a researcher’s stamp of approval. Like design, think of research as iterative. Take what you learned from one survey and apply it to the next one.

Sometimes “bad” data can be good data. You might realize that you could have added or rephrased a question. That’s great information to have moving forward. As much as we try, perfection isn’t always attainable, and that’s ok. Do the best you can, and take every opportunity to learn and improve. Build your surveys with integrity, report your findings honestly, and don’t be too hard on yourself.

Research, iterate, and repeat. Find scales that work for you and use them again and again.

Example survey

Parting thoughts

Consider how you might use this structure to measure your research objectives. Using a mix of closed and open questions can give you a jumping-off point for analysis, and richer contextual data.

There are pros and cons to any research methods you choose, and the fears folks have about unmoderated methods are valid. I believe that surveys are much more flexible and valuable than they often get credit for, though. The more surveys you design and analyze, the more confident you will be in your unmoderated research!

Interested in other articles like this? Check out...

• Your Step-by-Step Guide to Setting Up Unmoderated Generative Research
• Stuck? Here are 9 Ways to Analyze User Research for Meaningful Takeaways
• Three Approaches to Tagging that Bring Clarity to Qual Data