In my years working in UX, there have been times when I’ve received some pushback and questioning of user research. The standard, "You only spoke to 7/10/12 people?" is the most common question.

However, once I talk through the different sample sizes for methodologies, colleagues tend to accept the realities of qualitative research. But one day, I got a question that made me pause.

"How do you know user research is valid?"

It's easy to get caught up in the fast pace of tech companies, tight budgets, pressure from looming deadlines, and stakeholders' pet projects. While juggling the above, we may need to cut corners during our research projects.

However, we want to make sure these cut corners don't reduce the validity of our studies. Checking on project validity is an exercise I think every researcher should do to make sure we are still on the right track.

What is research validity?

Before we dive into the ins and outs of validity, let's go through a few definitions.

First off, whenever you hear about research projects, you generally hear the words: validity and reliability. These are two separate concepts, although they are very much intertwined:

• Reliability measures the likelihood of getting the same results if you repeat the test and ensures that the results are not entirely random. The primary way to increase reliability in a qualitative study is to include more participants or follow up with a quantitative component.
  
  
• Validity refers to what extent the chosen research method measures what it is supposed to measure.

Both reliability and validity are essential for a successful research project. For now, we will focus on validity.

Validity looks at what you are measuring, how you are measuring it, and if the study measures what you set out to measure. Lots of measuring, right? Since these are vague terms, let's look at an example:

A team comes to you and asks you to measure the usability of a photo editing app. You take the following steps:

1. Decide on usability testing, specifically looking at the time on task (efficiency), task success (effectiveness), and satisfaction
2. Choose the segment you need to test with and recruit seven people from that segment
3. Write the tasks for the study
4. Conduct the study and share results with stakeholders

Is this valid research? It depends on the details, of course, but overall it looks valid. This is because you set out to measure the usability and designed a study to give you that particular outcome.

Keeping validity in mind will also help you determine if the research question itself is valid. For example, can you measure if people prefer one design over the next through a qualitative research study? No, preference is not measurable with this method.

Or, can you measure how many croissants people will order from an online bakery in the next six months? Nope, because we can't predict the future.

Thinking about validity is a circuit-breaker in helping you understand if the question you are trying to answer makes sense and if qualitative research is the best approach.

Two types of research validity

Now that we understand validity let's go one step deeper. There are two types of validity in research projects:

Type 1: Internal validity

Internal validity measures how confident we are that the study results are trustworthy and not influenced by any outside (confounding) variables. Internal validity refers to the structure of the study and the related variables.

Type 2: External validity

External validity measures how generalizable your results are to other people, settings, or situations. External validity refers to how universal the results are.

Many things can go wrong with validity, and they are not all easily containable. We can never say, "yes, this is valid" or, "no, this isn't valid." Instead, we look at confidence. How confident are we that no other variables came in to confuse the results of our study? How confident are we that our results are generalizable to the broader audience?

These threats to validity come in all different shapes and sizes, and it's helpful to know what to look out for.

Threats to internal validity

Here are several ways internal validity can be threatened during a research study:

✔ Researcher bias

Leading or priming the participant for specific answers or actions that
causes them to react or respond in a way that is different than they
would have otherwise.

✔ Lack of consistency

Asking different questions to participants during the same study, or vastly different researchers conducting the same study.

✔ Attrition

People leaving a study early (ex: diary studies) which can entail a
biased sample of participants who chose not to leave due to a factor
such as higher motivation.

✔ Confounding variables

Something outside of the study that impacts or changes the results, such as the time of day you test with participants.

✔ Poor recruitment

Participants in the study have no context or are heavily biased toward the topic.

✔ Repeated testing

Repeatedly giving the same participant a task three times, they will likely learn to do better and answer differently.

Threats to external validity

Here are some ways that external validity can be threatened during a research study.

✔ Selection bias

The participants in the study differ substantially from the population or target audience.

✔ Hawthorne effect

Participants change their behavior and responses because they know they are in a study.

✔ Environmental factors

Time of day, location, internet connection, researcher bias, and other situational factors.

✔ Testing

A pre-test survey causes the participant to think more deeply about the topic and possibly respond differently.

Ways to increase the validity of your study

With all the juggling and questioning of user research, we also need to consider the validity of our studies. For a while, this felt like another obstacle to overcome. But, after some time, I tried to operationalize this step.

Here are some checks and balances you can put into place to assess the validity of your study:

Internal validity

If you want to improve the validity of your project, you will need to focus on the study design. Here are some factors that can help to improve internal validity:

✔ Varying the order of tasks

Consider changing the order if you conduct a usability test. At the beginning of the test, participants might make more mistakes as they adjust to the environment and the test material. At the end of the test, they might suffer from fatigue. Varying the order of tasks (or survey questions) will help negate these effects.

✔ Avoid bias

As much as you can, avoid asking leading or priming questions that can cause a participant to react or respond differently than normal.

✔ Randomize the order of designs

If you are testing several prototypes, ensure the order for each participant is randomized. For example, when I was testing three different prototypes, I made sure to vary the order I presented the prototypes.

✔ Study protocol

Follow specific steps during the study to mitigate confounding variables. For example, don't change or add questions after the second participant, which can skew the study results. Also, ensure that if there are multiple researchers on the same project, everyone uses the exact task wording and protocol.

External validity

✔ Sound recruitment

Recruit participants representing the population you want to study. Using a screener survey will help get the most appropriate participants.

✔ Use mixed methods

Follow up qualitative research with a survey or quantitative component to understand how/if your results generalize to your broader audience.

✔ Warm up with participants

Help participants get into the flow of the conversation by warming up with two to three questions outside of the topic of the test. Also, remind them there is no wrong answer.

✔ Try contextual inquiry or diary studies

To get a more "real-world" setting, try methods that get participants responding in a more natural situation and setting.

✔ Create scenarios

If you are in a lab setting, create a realistic scenario for your test that mimics something the participant would do or encounter in the real world.

Wrapping it up

Overall, looking into the validity of your research is hugely important and can help you increase the usefulness and trustworthiness of your results in many ways. It can help you:

• Control confounding variables that confuse or skew your results
• Determine that your insights are valid
• Focus on the right research questions that are applicable for qualitative research
• Understand how generalizable your findings are
• Determine which situations your insights apply to
• Figure out if you can translate to your results a larger population

If you can cite the validity of your insights, stakeholders will come to trust your results without question (or, at least, as many questions).

You may also like...

• The Ultimate Guide to Mixed Methods User Research
• A Simplified, Mixed Methods Roadmap: How to Marry Quant + Qual for More Insightful Results
• To Ask or Not to Ask? How to Design Research for Difficult Questions

‍

Nikki Anderson-Stanier is the founder of User Research Academy and a qualitative researcher with 9 years in the field. She loves solving human problems and petting all the dogs.

To get even more UXR nuggets, check out her user research membership, follow her on LinkedIn, or subscribe to her Substack.