Prove the Value of Your UX Research Insights with a Benchmarking Study

As researchers, we produce insights. These insights are a lighthouse for our teams to steer towards. They give teams a guiding path to creating functions, features, or products that solve real human problems.

Sometimes these ideas are a massive change to the current product. To our stakeholders, it can feel like a scary gamble.

If you're in this position, you may encounter this seemingly simple question, "How do you know your insights and recommendations are improving user experience?"

The first time I heard that question, I was tongue-tied. “Well, if we are doing what the user is telling us then, of course, it is improving.” Let me tell you that didn’t cut it.

Then I learned about benchmark studies.

These studies allow you to test how a website or app progresses over time and where it falls compared to an earlier version, competitors, or industry benchmarks. Benchmarking will allow you to answer that above question confidently.

Jump to:

• How do I conduct a benchmarking study?
• How often should I be running benchmarking studies?
• What metrics should I be using?
• What do I compare my product to?

How often are new iterations or versions being released? How many competitors do you want to benchmark against, and how often? Do you have the budget to run these tests on an ongoing basis? Benchmarking tests cost money and effort. You have to recruit and incentivize participants. Whenever possible, I recommend doing them in person, but you can also conduct them remotely.

What are we trying to learn?

Do we want to compare a new version of your product with an older version? Analyzing two different versions will see how the insights are impacting users. You can also compare your product to competitors or an industry benchmark (note, this isn't always possible). Answers these questions will help you determine if benchmarking is the right methodology for your goals.

What are we trying to measure? 

What parts of the app/website are we looking to measure—particular features or the overall experience? It is usually fruitful to test the entire product with benchmarking tests, rather than just smaller flows.

✔ Write an interview script

Once you have your goals and objectives set out, you need to write the interview script. This script will be similar to how you write usability testing scripts. The questions focus on the most critical and essential tasks on your website/app. There needs to be an action and a final goal. For example:

• For Amazon, “Find a product you’d like to buy.”
• For Wunderlist, “Create a new to-do.”
• For World of Warcraft, “Sign up for a free trial.”

Notice, these tasks are incredibly straightforward. Don’t give any hints or indications on how to complete the task, as it will skew the data.

I know it can be hard to watch participants struggle with your product, but that is part of the benchmarking and insights you can bring back.

For example:

• “Click the plus icon to create a new to-do.” = Bad wording
• “Create a new to-do” = Good wording

If you would like to include additional questions in the script, you can use follow-up questions, asking them to rate the difficulty of the task.

After you complete the script, and once everyone has input any suggestions or ideas, keep it as consistent as possible. It is challenging to compare data if the interview script changes.

✔ Pick your participants

As you are writing and finalizing your script, it is good to begin choosing and recruiting participants.

Although standard user research studies, such as qualitative interviews or usability testing, generally call for fewer participants, we must realize we are working with hard numbers and quantitative data.

It is ideal to set the total number of users to 25 or more. At 25+ users, you can more easily reach a statistical significance and draw more valid conclusions from your data.

Since you will be conducting studies regularly, you don’t have to worry about going to the same group of users over and over again.

It would be beneficial to include some previous participants in new studies, but it is fine to supplement that with new participants.

The only important note is to be consistent with the types of people you are testing with—did you test with specific users of your product who hold a particular role? Or did you do some guerrilla testing with students? Make sure you are testing with those users for the next round.

If you are testing with around 25 users each time, how many times can you realistically test with your budget?

How will this relate to other product development?

If you're releasing updates on a more random basis, you could develop ad-hoc benchmarking studies that correlate to releases. However, this may not be the most effective way to show data.

You want to see progress over time and how your research insights are potentially improving the user experience. Determine with your team and executives the most impactful way to document these patterns and trends. Just make sure you can run more than one study, or the results won't be actionable!

The team at T-Mobile consistently needed robust insights on a tight timeline. See how they were able to shift from a reactive to a proactive research strategy, gather in-context mobile data quickly, and get their stakeholders more engaged with the data.

It’s not always clear whether the data you've gathered is statistically significant. These methods will help you navigate that question.

"Should I report a one-off Insight?" Though one participant’s opinion isn’t statistically significant, there are times when a piece of feedback may point to a larger issue.