As researchers, we have all heard someone pipe up and say, "But you only spoke with seven people!" Sigh. Eye roll. Clenched teeth.

It is not your stakeholders’ fault that this question comes up. Companies are asked to be “data-driven.” For many, that translates to backing up facts with large samples. Statistics and data have existed much longer at companies than qualitative research and patterns in conversations.

So what happens when you are on the receiving end of this question? I've gone blue in the face trying to explain: “We only need to talk to X amount of people before the findings become repetitive." However, it doesn't seem to stick.

Sometimes, the most effective thing to do is to strengthen the perceived validity of our qualitative data with “the numbers.” Now, I will be cautious not to say, "quantifying qualitative data." Qualitative data should not (and can not actually) be quantified; quant data is a separate entity. However, we can share trends we found in qualitative research and pull in quantitative data can help “back them up.”

Some quick and easy ways to validate qualitative findings:

I understand why people are hesitant to make vast generalizations about a product, company, or feature based on seven people. How can we take action to make stakeholders feel better about the “danger”?

Before the research:

1. Choose the right method. Thoughtful research techniques will help to assure validity in qualitative research. Make sure you are considering the research problem and the best methodology to support it.
2. Recruit the best participants. If you are talking to the wrong people, you’ll get false or incorrect data. Ensure you are talking to the most relevant participants for the study.
3. Read the fine print of the 5-7 users rule. You’ve probably heard the rule of thumb: “You only need 5-7 users” for valid findings. There is a catch with this rule that some people can miss. We can't merely test with any 5-7 users and call it a day. This rule only applies when we interview 5-7 participants per persona or target group. Organizations generally segment users in some way. Make sure you are speaking to at least 5-7 users per each segment.

Qualitative data should not (and can not actually) be quantified; quant data is a separate entity. However, we can share trends we found in qualitative research and pull in quantitative data can help “back them up.”

Nikki Anderson

After the research

After the research is when most of the real validation can take place. I have used the following methods to help stakeholders, and teams, feel more confident in making decisions based on user research findings.

Have multiple perspectives during the analysis

A great technique to establish validity is to seek alternative explanations of the research results actively. If you are researching in a vacuum, you only have your own perspective. By bringing others into the mix of analysis, you get a more diverse range of interpretations. Bonus points if you can include people who have been at the company longer than you and those who just joined.

When multiple perspectives are going into the analysis, you are less likely to interpret or assume incorrectly. You can come to a consensus on what each finding means. If you can exclude other possible scenarios or interpretations, you can strengthen the validity of the insights.

Surveys

There are two surveys I commonly use after a qualitative research project to get numbers behind my insights:

Unmet needs surveys

The unmet needs survey is my favorite survey to use for validating qualitative data. This survey comes from "Outcome-Driven Innovation" and "Jobs to be Done,” but you can use it to back up findings from any methodology.

To create the study, you take the “user needs” you gathered from your qualitative research and craft each need into a statement. Then, in your survey, you ask: 1) “How important is meeting this need to you?” and 2) “How satisfied are you with the current ability to meet this need?”

Each statement should have:

• A direction of change. How can the situation improve for the user? Often this includes words like "maximize," "decrease," "lower," or "increase."
• A success metric. What is the thing that needs to change from above? It usually has to do with time, effort, or skill
• An object. What is the need? What will be changed and measured?

As an example, let's say you heard two needs in particular:

• I need to be able to know the restrictions of bulky luggage on a train.
• I need to be able to share the cost of a trip with friends or family, so they don't have to pay me back later.

Now you create statements based on these needs. Remember: direction of change + success metric + object

• Minimize the time it takes to find train restrictions for bulky luggage.
• Reduce the likelihood of not being able to bring specific luggage on a train.
• Minimize the time it takes to receive payment for a trip from family/friends.
• Reduce the time it takes to split trip costs among friends/family.

Feature request surveys

I am careful with feature request surveys but must admit I do utilize them in certain circumstances. People generally don't know what they want, so I never take exact feature requests from a user interview.

Instead, I use this when the team has a problem to solve and has several feature ideas. After idea generation, I will use a survey to ask users which ideas make the most sense. This survey helps the team focus on two of the top solutions to then test with users.

For example, you hear many users are struggling with properly budgeting a vacation. Your team comes up with several ideas, such as:

• Sort destinations by price
• Search for destinations by maximum budget
• Show groupings of trips by price
• Filter destination tips by budget
• Show budget destinations on the homepage

Instead of working on prototypes for all these ideas, or deciding internally, you can create a survey to ask users to pick which is most important to them.

SEQ or SUS

These two measures, the Single Ease Question (SEQ) and System Usability Scale (SUS) are a great way to see where the user experience is breaking down. For instance, during qualitative research, if we heard that the check-out experience is painful, we could place one of these surveys in the flow. This survey could help gauge, on a broader scale, if other most users are feeling the same.

Analytics

Using whatever form of analytics your organization is gathering on user behavior is a low-effort and high-impact way to validate qualitative findings. This is especially true of usability testing. If you find, in your usability test results, that users are struggling with completing a form, you can use analytics to prove your case.

Open the company's analytics platform and look into the number of errors on that form, click-through rate to the next page, or bounce rate. In my experience, usability tests usually indicate a more significant problem that you can find in analytics.

Beta testing

In the research world and outside of product/tech, this is called respondent validation. This technique involves testing initial results with participants to see if they still ring true. Beta testing is an excellent way to initially test findings and ideas with a wider (but small) audience.

Your research findings become more reliable if responses become more consistent across a more substantial number of participants. You use beta tests to see whether they are using the product/feature, how they are using it, and what bugs, improvements, or innovations come up.

If beta testers aren't using certain features, chances are your actual user base won't be either. Beta testing can help you see how a product or feature would react in the wild, and it is a great way to test with a small but valid sample size.

‍

Nikki Anderson-Stanier is the founder of User Research Academy and a qualitative researcher with 9 years in the field. She loves solving human problems and petting all the dogs.

To get even more UXR nuggets, check out her user research membership, follow her on LinkedIn, or subscribe to her Substack.