UX Research: A Practice or a Science?

I was once working on a freelance project with other researchers. I'm not talking about user researchers, but academic researchers. They spent years in the field building experiments and tests.

They had published papers on an online platform for students or other practitioners to read. I was there to help them understand their user base and the mental model of the users finding and reading scientific papers.

The conversations started strong with words like "research," "participant," and "methodology." But then I started explaining the user research process, using these same words. I mentioned speaking with around 15 users from two main segments (or personas). I then pitched purely qualitative methodologies in the form of 1x1 interviews and contextual inquiry.

Their eyebrows could not have crawled further up their foreheads.

After additional discussions, they resigned to my ideas. I was, of course, "the expert" as someone in the meeting said. But that meeting made me realize something:

We are saying the same words but speaking a different language.

User research has the word research tucked right in there, no matter what you call it: user research, user experience research, product research, design research, ethnographic research. When people think about research, they think about science. However, let us look a bit closer to the differences between science and user research.

The Purpose

Scientific research

The purpose of scientific research is to understand the nature of our physical universe. When I think about the hard sciences, such as chemistry, biology, and physics, I conjure up images of experiments in a lab with beakers and lab goggles. This image feels very applicable to the above definition.

This hard science uses systematic observation, experiments, and sometimes mathematics to gain or add to existing bodies of knowledge. In hard science, the results need to be reproducible. If you conduct the same study a second time, you must find the same results.

The above definition does not feel like user research in the slightest. At least, not to me. But let's dig deeper.

We regard user research as more of a social science, like psychology and anthropology. It’s the scientific study of human society, the nuances of human behavior, and social relationships.

Many people coming from these fields find user research as quite a natural fit. However, social sciences still go through the scientific method when researching, and the points of validity and reliability are hugely relevant.

The purpose of both hard science and social science is to test a process or concept and generalize your findings to a much broader population.

User research

I have observed that user research doesn't go through these rigorous processes. I can say that, if I continue usability testing the same product with 150 versus ten users, the same trends will continue to surface, but we will never get the same results.

When you think about it, user research has a different set of objectives than scientific research. It aims to help avoid pitfalls in development and tech projects. In a tech organization, people can tend to base ideas, designs, concepts on their own needs, desires, or preferences. They make decisions based on gut instinct. User research opens the door to shattering those biases and listening to the people who are using a product. It highlights blind spots the company might be missing for innovative ideas and empowers teams to make better decisions.

I believe a company wants to generalize user research findings to a broader audience. However, the results of user research are not direct and conclusive findings. As user researchers, we find trends and patterns that point to ways we could improve or problem areas by creating a solution.

Approach

Scientific research

Scientific research, both in the hard and social sciences employ the scientific method. It is a rigorous process that can take many years, including peer reviews and study replication. Often, the desired outcome of a research study is a published paper containing empirically testable findings.

The scientific method is something I learned a lot about in my graduate program. Even coming from the social science of Psychology, it was not only expected but required that we use this method in our research. I can still remember it getting drilled into my brain:

1. Ask a question
2. Research existing sources
3. Form a hypothesis
4. Design an experiment/study
5. Conduct research
6. Draw conclusions
7. Create a report

I found using the scientific method quite tricky. Social sciences are mainly concerned with complex issues that require sophisticated, but often less clear cut approaches. These approaches may be less satisfactory but more realistic due to the problems we investigate.

Even though it might seem similar to some of the steps taken in user research, the pure scientific method is an arduous process to use. I conducted a study on how anxious people felt when looking at anti-smoking or anti-obesity ads and then spoke to them about their behaviors after viewing the ads (similar to a diary study). In user research terms, this would take maybe four weeks to complete end-to-end research. It took months to secure a budget in my academic setting, get through IRB approval, and write a (very lengthy) paper.

User research

User research takes a different approach than the exact scientific method. The way I explain user research is more of a scientific inquiry. We are less methodological and, instead, are more exploratory and fast-paced (at least, in qualitative research).

User research approaches projects through ideas from stakeholders, users, or the researchers themselves. These concrete ideas give us a narrow focus on a particular product or service, or a problem space we want to explore.

The ideas come in fast and furious. We aren't in the same academic setting where research does (and can) take months or years to complete.

In the product world, we are in a high-speed environment. We need user research completed yesterday, or last week. We have to adapt to work in cadence with development times, product launch dates, and expectations from stakeholders. This difference in the environment is significant to consider as we criticize user research for not being more scientific.

User research methods mostly pull from the social sciences, such as qualitative interviews, contextual inquiry, mental models, and field studies. When we employ these qualitative research, they are, similarly, with small sample sizes. We then generate hypotheses that can be tested through broader quantitative analysis to fulfill the validity and reliability aspects of our practice.

This, of course, is not perfect. I cannot say that all my qualitative studies have no sampling bias or subjectivity.

The ideal outcomes of user research are often lightweight documents or artifacts intended to produce just enough information. The deliverables need to be immediately actionable. To create these deliverables efficiently, we need to move fast with our analysis. Often, we don't have more than a few days to synthesize the information. Sometimes we are doing it alone, which can lead to biased results.

What can we do?

Let's not try to make user research something it’s not. I don't want to stuff user research into the scientific method box and condemn any future studies that don't apply that same rigor. It simply is not realistic.

Some ideas I will take forward in my practice are:

• Continuing to educate myself on proper experimental design
• Using the parts of the scientific method that make sense for research (ex: really thinking about the design of the study, using hypotheses more)
• Finding ways to prove reliability and validity in my studies (such as through quantitative methods)
• If research feels too rushed, push back on stakeholders
• Apply more rigor to reduce sampling bias in my studies
• Always include at least one other person in analysis and synthesis of research

Remember: not everyone has had scientific method training, and not every UX researcher knows how to design a perfect experiment or study. Instead, we need to find a way to balance the crucial parts of science with user research's realistic nature.