Obsessed with understanding what makes People tick

Dnd

The Simple Way to Design a Standout UX Research Survey

Surveys are one of the most reliable tools in a researcher’s toolbox. Here’s how to design one to help validate your qualitative insights.

Survey design is complicated.

For me, I dread putting surveys together. The reason I don’t look forward to creating surveys is simple: people have high expectations of the information they will glean from them, and they rarely live up to these expectations.

Surveys are too often used to get answers to questions better suited to other methods, such as qualitative interviews. At times, people use questionnaires as a cop-out. Because of this usage, I often steer people away from them.

Sometimes, though, surveys can be helpful for a study. My favorite way to use surveys is as a supplement to other methodologies. If necessary, I will use them in isolation. I only do this when I have no time or budget to use them in conjunction with other research techniques.

Regardless, surveys are challenging to design. Generally, during user research, you are trying to get at a deeper meaning behind someone’s actions or motivations. With this method, you are attempting to ask people to answer questions with a predetermined list of answers. There is no probing, digging, or explanation happening during a survey. If you ask for a reason, you aren’t always guaranteed to get one. At least not the deepest level of description.

It is necessary to define the goal and objectives of the survey. By brainstorming goals and objectives, you are better able to design the study that will get you the type of results you will need, and mitigate everyone’s expectations.

People have high expectations of the information they will glean from [surveys], and they rarely live up to these expectations.

Defining a goal:

Before sitting down to write questions, I always create a goal for the survey. I do this by asking myself the following questions:

  1. What are we trying to learn about our users?
  2. How will this survey help us?
  3. What do we expect from the results (assumptions)?
  4. What do we expect to do with the results?

For instance, we could be trying to find out the answer to, “What new features should we prioritize next?”

With this survey, we want to be able to understand which new features we should start working on that our users would enjoy. We want to be able to take these results and prioritize our work.

So we would generate a list of different answers. Ideally, these answers are based on past research insights. Then, we leave an “other” option for users to write in their response if they want to.

The survey might look like this (for an online board game platform):

Q: Which feature would you love to see us work on next?

  • Worldwide online competitions
  • Customizable boardgame characters
  • Shareable record chart of wins/losses
  • AI boardgame challenges (play against the computer)
  • Practice tournaments
  • Other: open-text

It is essential to understand that we are not trying to figure out why a person wants a particular feature. Ideally, we have done the qualitative interviews in which our users explained what they needed and why.

We are trying to get a larger sample size to prioritize better what to work on next. Surveys don’t give you the why; they give you the what.

Typically, surveys are a bit more complicated than the above example and involve many more questions. In this case, there are many more pieces of the puzzle to consider. You have to think about question structure, number of questions, flow, and many other considerations. A longer survey requires much more brain power and time.

Surveys don’t give you the why; they give you the what.

Before we get deep into it, here are some quick and easy tips you can apply to your next survey:

  1. Don’t use UX jargon (ex: ask about frustrations instead of pain points)
  2. Ask one question at a time
  3. Use simple and easy-to-understand language at an eight-year-old reading level
  4. Give the “other” option whenever possible
  5. Test your survey before giving it to users
  6. When possible, use a progress indicator
  7. Use open-ended questions as much as possible, but not for every single question
  8. Make use of logic functionality
  9. Leave room for feedback by allowing people to comment on the actual survey
  10. Use prior research to give you multiple-choice options

The dark side of surveys

Now that we have gone over the easier side of survey writing, let’s take a look at the dark side. The dark side happens when we want to understand more than an answer to a simple question.

It becomes much harder when we want to use surveys to understand behavior or opinions. However, it is not impossible. Again, I do recommend doing qualitative research in addition to sending out a questionnaire or poll.

To illustrate this, let’s take another example. The online board game platform was so successful that we decided to start an entire online board game store and community. We are looking to understand the following from our survey:

  1. How people would like to purchase board games online
  2. What people expect of a board game community
  3. What they think of other online board game communities
  4. Get their opinion on a prototype of the online store

That is a lot to ask of a survey, but I am going to the extreme for this example. I would hope, alongside or before/after, we would conduct some qualitative user research to understand our potential users even better. Before I sent out this survey, I would include a screener that asked if people were board game users and if they had purchased board games online in the past. This would help ensure I get helpful answers from relevant participants.

I will always ask about past behavior before asking about anything future-based. Gathering people’s past experience is my biggest piece of advice for getting the most relevant and insightful information from users. Always ask about the past before the future because past behavior predicts future behavior.

Questions for understanding behavior:

  • How do you prefer to purchase board games?
    • In-store
    • Online
    • Other
    • Logic: why do you prefer to purchase board games that way? (open text)
  • With which stores have you purchased board games online?
    • Multiple-choice with “other” option
  • How often do you purchase board games online?
    • 1-2 times a month
    • 3-4 times a month
    • 5+ times a month
  • Why do you purchase board games online?
    • Either: multiple choice with answers from previous research and “other” option OR open text
  • What is your main goal when visiting an online board game shop?
    • Multiple-choice with other
  • Is there anything you look for on an online board game store that is missing?
    • Yes/No
    • Logic, if yes: what are you looking for?
  • Have you ever returned a board game you purchased online?
    • Yes/No
    • Logic, if yes: Why did you return it? (open text)

Questions for gathering opinions:

  • Think about a time when you purchased a board game online. How was your experience?
    • Scale: Very good, good, neutral, bad, very bad
    • Why (open text)
  • Think about a frustrating experience you had purchasing a board game online. What happened to make that experience frustrating?
    • Open text
  • What is the most difficult part of buying a board game online?
    • Either: multiple choice with answers from previous research and “other” option OR open text
  • What do you look for when going to an online board game store?
    • Open text
  • What would make you purchase a board game online?
    • Open text
  • How would you improve the experience of purchasing a board game online?
    • Open text
  • What do you like about purchasing board games online?
    • Either: multiple choice with answers from previous research and “other” option OR open text

Questions for prototypes:

  • What are your initial impressions of this website?
    • Open text
  • What is the most important information on this page?
    • Open text
  • What information is missing?
    • Open text
  • What information is confusing to you on this page?
    • Open text
  • What would you change to improve this page?
    • Open text
  • Would you use this page?
    • Yes/No
    • Logic: why or why not?

Here we have a mix of open-ended and closed questions that touch on behavior, opinion, and prototype. If I were offering an incentive, such as a gift card or discount to the store, I would ask users all of these questions. If I could not compensate, I would try to pick and choose 7-10 questions from this bank.

Once you feel comfortable with wording, you can use different question types, such as matrix or forced rankings. I stuck with multiple-choice, scale, and open text because they are the easiest to start with. I recommend exploring other options as well.

Now that you have your questions…

Here comes the much more fun part of the entire process: sending out the survey. If you don’t already have a tool for surveys, here are three of my go-to’s:

Typeform:

  • Beautiful UI
  • Easy to use
  • Mobile version
  • Logic flows
  • Can be expensive for smaller teams

Google forms:

  • Free
  • Various question types
  • Logic flows (although difficult at times)
  • Not as intuitive
  • Not as beautiful

HotJar:

  • Easy to use
  • Good UX/UI
  • Logic flows
  • Can be expensive

Then it is finally time to send your survey out to potential respondents! If you are posting using a tool, then I try to use a catchy title. It helps a lot if you can offer compensation for taking the survey. If so, I always include that in the title, such as: “Help us improve for a 15EUR gift card.” If you aren’t able to offer any money (and that is okay), you can try titles like: “X Company needs your help!” or “Help us improve for you!” Once you get them with a title, always make sure to include a small description of the survey topic, as well as an approximate amount of time it will take. Include an email address for any comments or questions.

If you are sending the survey in an email, you have some more room for explanation. I use the following template with a subject line similar to what I mentioned above:

Hey [name], (if you don’t know the person’s name, you can say: “Hi there!”)

We’re working on [the topic and goal of research]. It would be great if you could complete a quick survey — it will take you [average time] minutes. In exchange for your time, we will offer you [compensation]. We will use your feedback to improve our product and make it more useful for you. Thanks for your help!

Best,

[your name and role in a company]

Voila - you have sent out your survey! What do you do now? Wait for the responses to roll in. If they aren’t, make sure nothing is broken and that you asked the right people for the questions you wanted to be answered. Also, check to see if the survey is too long, especially if you aren’t offering an incentive. 

Brownie points if you respond to each user who took the time with a thank you and explain how their participation helped you.

Nikki Anderson

Nikki Anderson is a qualitative user experience researcher with about 5 years in the field. She loves solving human problems and petting all the dogs. Read more of her work on Medium.

Curious as we are about what makes people tick?

Get new People Nerds articles in your inbox.