How to Write (Better) Survey Questions: Systems for Collecting Clean and Meaningful Data

Writing survey questions can be difficult—for many reasons.

You don't want to be so vague that no one understands how to answer your question. You also don’t want to be so specific that people feel forced to respond in a way that may be untrue. You also don't want all of your questions to be open-ended so that no one takes the time to answer them meaningfully, but you also don't want them all to be closed questions, which leaves out the possibility for people to answer them truthfully.

Where is the balance, and how do we achieve it? How do we construct the best survey questions possible?

First, we should talk about when surveys are most useful. Sometimes, it’s better to do qualitative research rather than building and sending out a survey. I generally use surveys in these four scenarios (which are not inclusive of all scenarios):

1. Providing an understanding of the direction of your next qualitative research project should go in
2. Validating/disproving hypotheses from qualitative projects
3. Honing in on specific concepts that come out of other projects
4. Receiving specific feedback on features

Now let's jump into how to construct survey questions and flow.

Back to basics: writing survey questions

When writing a survey, every part of it must contain the most optimal information possible. With this in mind, we need to choose the best techniques to construct the best questions.

There are different overall types of survey questions:

Closed questions. These prompt participants to answer from a fixed number of possible responses. Examples of closed questions include:

• Yes/No questions
• Multiple choice type questions (radio - only choose one option)
• Multiple choice type questions (checkbox - select more than one option)
• Rating questions

Open-ended questions. These allow users to respond however they want. Examples of open-ended questions include:

• Stand-alone questions
• Follow-up questions
• "Other" option on a multiple-choice question

Choosing which one to use depends on the type of data you are trying to get. Here are the advantages of each:

Closed question:

• Have a higher response rate
• Are easier to analyze
• Can provide statistical significance
• Can be used to "quantify" qualitative findings

Open question:

• Can provide insight into "why" users are answering in a specific way
• Allows for responses you may not have expected
• Are much more abundant in qualitative detail
• Gives respondents the chance to express themselves fully

The best advice: have a mix of closed and open-ended questions. I typically use the rule of 60/40. Sixty percent are closed questions, and forty percent are a mix of the different types of open-ended questions. Forty percent may seem like a lot of open-ended questions, but not each one is a stand-alone.

Types of Closed-Ended Questions

Now we can dive into more specifics and when they are most useful. Since open-ended questions are more self-explanatory (and you can read more about them here), let's focus on closed-ended question types.

Types of closed-ended questions:

Likert scale questions. This type of question is a statement where participants can rate their level of experience or agreement. There are two main characteristics of a Likert scale:

1. There is an odd number of responses, which allows for a neutral response
2. There is a spectrum of feelings on which a user can rate a level of agreement or experience

For example: After using the X product, I found the navigation to be [1 (very bad) to 7 (very good)]

Semantic differential scales. This type of question pairs opposite adjectives at either side of a scale. It helps you understand the emotional attitude of a respondent towards a concept, organization, product, or service.

For example: Love - - - - - - - Hate; Powerful - - - - - - - Weak; Likely to return - - - - - - - Unlikely to return

Multiple choice questions. The most common type of question where a respondent is presented with a question and has to choose from a predetermined list of questions.

For example: How many times did you exercise last week? Zero, One, two, three, more than three, Other

How to structure great survey question

With this knowledge of different question types, how do you construct the best survey questions possible? There are a few different ways to make sure your survey questions will be as answerable as possible.

Keep it simple. The average literacy rate is around the 5th-grade level, so use direct and straightforward language. This means avoiding industry jargon, advanced wording, or unknown abbreviations. If you must use complicated words or abbreviations, be sure to include directions or explanations on what they mean. Also, keep the questions short. The fewer words, the more direct and more easily understood they will be.  

Leave out loaded questions. Loaded questions make many assumptions about a person's feelings or context. They force the respondent to answer in a way that may be untrue and could cause drop-out or skipping. Avoid these questions by introducing logic to your survey.

For example:

• Loaded question: "Where do you enjoy drinking beer?"
• Unloaded question sequence: "Do you enjoy drinking beer?"; If yes, "Where do you drink beer?"; Follow-up: "Is there a particular place you enjoy drinking beer?" If yes, "Where?"

Avoid double-barreled questions. Double-barreled questions can be challenging to spot because we naturally ask them in conversations. A question like this includes asking two questions at once, which makes it impossible for a participant to respond appropriately. Go through each item and ensure there is only one answer, and be sure not to include and/or questions.

For example:

• Double-barreled question: "How do you feel about the information and/or functionality on this website?"
• Two separate questions: "How do you feel about the functionality of this website?" AND "How do you feel about the information on this website?"

Dodge double negatives. A double negative refers to the presence of two negatives in one statement or question. These types of questions create an unnecessary amount of confusion in the mind of the respondent. It also makes it nearly impossible for the researcher to determine what respondents were agreeing or disagreeing to accurately. There is a simple fix for this problem; just take the negative out of the sentence!

For example:

• "Do you disagree that dogs should not be allowed in restaurants?"
• "Should dogs be allowed in restaurants?"

Include 'I don't know' or 'Not applicable' options. These options prevent people from skipping questions or giving false answers. People get angry when asked questions they can't answer honestly, and it skews your data if they try to.

Avoiding bias and leading

I wanted to include an entirely separate section for avoiding biases and leading because of how important it is to avoid these questions. With a survey, you can't backtrack or rephrase what you say. Participants will take what they read and do their best to answer it. To get clean and meaningful data, we must make sure we aren't leading or biasing our respondents in a particular way. Here are some examples of leading and influencing, with corrections:

Leading: How great was our coffee?

Fixed: How would you describe our coffee?

Leading: What problems do you have with the sales team?

Fixed: Describe your experience with the sales team.

Priming: "Please rank all of the following electronic products we offer" and "Does our website offer enough electronic products?"

Fixed: Do not list all of the products you offer and then ask participants if there are enough. You just listed a lot of them so, they are more likely to answer that there are enough, even if they aren't what the participant is looking for

Survey flow

All of this works even better when you have a seamless survey flow and structure. Here are some tips to keep in mind when structuring a survey:

• Include an introduction. Give them an idea of what you will be asking them, how much time it will take, and how to get in contact if they have any issues. The introduction is also an excellent time to include the incentive if you are offering one.

• Keep the survey short. Once the amount of time it takes for someone to complete a survey hits 7–8 minutes, completion rates dropped by 5% to 20%. The highest response rates come when surveys are quick and painless. And the best way to reduce users' time and suffering is to reduce the number of questions.

• Use "the funnel." Start with broad general questions that are easy for the respondent to answer. Move on to more complex questions in the middle, and then return to general questions at the end. This sequence allows respondents to warm up and get involved in a survey before the more thought-provoking questions start.

• Vary the order. Make sure to change the order of the survey responses on multiple-choice questions. When selecting from a list, many people choose the first thing that sounds like it might be right and go to the next question.

• Always practice your survey. Before sending it out to respondents, make sure to look it over and try it. Also, send it out internally to your company, or friends, to get a few pairs of eyes on it. Double-checking the survey makes sure there aren't overwhelming mistakes and that it isn't overburdening your participants.