How to Optimize Unmoderated User Tests
You can't be everywhere at once. That's why unmoderated user testing might be just the thing to scale your research.
Unmoderated user testing gets a bad reputation. I refused to do unmoderated testing for years and would always preach against it. I would tell others how horrendous unmoderated tests were, what a waste of money they could be, and that the data was virtually useless.
I held a serious grudge.
Now that I'm much further along in my career, I have learned precisely why unmoderated user testing can be so effective—and how to discern when it's the right approach.
Jump to:
What brought me to unmoderated testing
A few years after accepting a position, I was still the only user researcher at a company. Fortunately, my colleagues were super interested in user research. I had research requests coming out of my ears. Some of the asks were for generative research, others for usability tests, and some card sorting. There wasn't enough time in the day, or a week, to get everything done.
I started to include multiple usability tests in one session, and soon I was juggling up to three tests at once. It wasn't realistic, and I felt like I wasn't getting enough information for individual teams. I was a mess. I knew something needed to change when I went into a usability test entirely unprepared. I was as confused as the participants when things did not work as expected, and I forgot the hotspots and flow.
I went back to the drawing board. How could I engage and help all these teams impactfully and still keep my sanity? I had to admit to myself that it was time for unmoderated user testing.
What is unmoderated testing?
Unmoderated usability sessions are user tests that the participant completes alone. You build a test similar to a moderated user test, with flows, hot spots, and tasks. However, you then send the link to the participant and have no real-time interaction with them.
The participant then goes through the flow and tasks whenever they have time, and you get sent the recording. With specific tools, you can email questions to the user after the session.
Unmoderated tests are not monitored or guided by a moderator. The participant completes any tasks and answers questions at their own pace, on their own time at location of their choosing.
Why you should use unmoderated user testing
As I looked into unmoderated tests more, I saw why others had chosen this method in the past:
✔ Get more responses
You can get many more responses in a shorter time since you are sending a link that multiple users can access at once. Gaining more answers is very helpful for generalizing findings across a larger population. It can also help with teams wary of small numbers that come with qualitative research studies.
✔ Scale up research
They are great for when you're in a rush to get users. If you're juggling many studies requiring you to be a moderator, these tests can get answers while conducting other research.
✔ Collect more data
You collect statistically significant amounts of data that may give you higher confidence in findings. You can segment these tests into different areas (ex: location, age, gender) to see the differences across demographics.
✔ Recruit more participants globally
You may be able to recruit more participants since users can complete sessions whenever they want. You can also access many more types of participants across the world since unmoderated user tests are always remote.
✔ Free up your time
It's easy for other stakeholders to independently set up unmoderated tests (once they learn how to write tasks correctly). By giving them this task, you can free up your time even more.
Usability in action!
We talked with Design Researcher Meghan Earley about how Dropbox met user needs by going beyond standard usability testing—leveraging video and diary studies to look longitudinally. Check out their field report and see how they were able to gain more confidence in user issue reports and see increased investment from their stakeholders.
When to use unmoderated user testing
Like any other user research method, there is a time and a place for unmoderated user testing.
✔ Gather feedback on a few specific components or changes
Unmoderated tests are best for getting input on minor UX or UI changes. Since you cannot explain a complex flow or feature, it's better to use unmoderated user tests for specific elements.
✔ Identify small issues on a live website/app
If you have a screen you want feedback on, unmoderated user tests can be a great friend. Participants can identify minor issues in the UX on a live website or app through specific and straightforward tasks. Keep in mind, you can also use prototypes for unmoderated tests, but they have to be significantly simplified. It's always easier to run unmoderated tests on a live website or app.
✔ Discover if users can find relevant product information
An excellent use for unmoderated tests is to understand if participants can find information on your website/app.
✔ Stay on a tight budget
If you are struggling with budget and also have the right goals, running unmoderated tests can be cheap. Running 25 unmoderated tests can be less expensive than running six moderated usability tests.
✔ Obtain a large sample size
Sometimes teams are wary of large sample sizes, especially if the change can have a substantial financial impact. Just keep in mind the tasks and changes have to be small. However, I have run into situations where a small change could significantly impact the organization. When teams want more confidence in decisions, especially for different segments, unmoderated testing can be a gem.
✔ Determine if users understand the value proposition of the product
You don't have to sequester unmoderated tests to usability feedback. These tests can also give you feedback on your brand and the potential value your organization brings to users.
✔ Collect feedback quickly
We all know about tight deadlines. User research can sometimes get squeezed into the end of a project, causing the timelines to be short. You can run an unmoderated user test over a few days (and the weekend), giving you findings much quicker.
✔ Don't expend emotion or imagination
When I say simple tasks for unmoderated tests, I mean it. Use unmoderated tests for tasks that don't need much emotion or imagination from the participant. The tasks should be simple and easy to follow for the participants.
When NOT to use unmoderated user testing
If you are looking to better understand the following situations, I would advise against unmoderated user testing.
✘ Complex topics
If you have a complex topic of testing, unmoderated user tests are not the way to go. You're unable to ask follow-up questions or guide the participant through complicated flows or features. Instead, try moderated usability testing.
✘ Early prototypes
Early-prototype testing is difficult without a moderator to explain and help participants recover from errors or limitations of the prototype. If participants can't understand how to use a prototype, they're likely to drop out or skip tasks, rendering the data unusable. Stick to live websites and apps or extremely simple prototypes that don't require much interaction. For this goal, I recommend moderated usability testing.
✘ Deeper understanding
If you need to understand a particular topic deeply, moderated tests are significantly better. Since there is no chance to ask "why" or dig deeper during unmoderated tests, you'll get more shallow answers. For this, generative sessions (either remote or in-person) would be more helpful.
✘ Strong emotional reactions
Looking for strong emotional reactions from participants is not a great goal for unmoderated user testing. Participants tend to be less engaged without a moderator, so they generally don't emote as much as they would otherwise. If you want strong emotional reactions or imagination, try moderated usability testing.
How to optimize your unmoderated user tests
There are some ways to optimize your unmoderated user tests to ensure you get usable and helpful data. Whenever I set up an unmoderated test, I always go through this checklist.
✔ Prompt participants to constantly think out loud during the test
At every screen, remind participants to think out loud. It may seem repetitive and annoying, but you aren't there to nudge them when they go silent.
✔ Keep it simple
The unmoderated test needs to be as simple as possible. Provide enough detail for the participant to complete the task independently and include any information they need to finish the task. For instance, if you need to test a check-out, provide false credit card details for the participant. You want to remove as many blockers as possible.
✔ Have many colleagues review the test
Fine-tune every instruction, task, and question to eliminate the potential for misunderstanding. Participants can easily misunderstand a task or instruction. By getting colleagues to review the tasks, you can check if there is a shared understanding, which increases the likelihood of participants completing the test correctly.
✔ Do a dry run see what it's like for the participant
I always do dry runs before a study begins, even with unmoderated testing. By trying out the tool and experiencing the other side, you can iron out any issues before the participants see the test.
✔ Over-recruit participants
No-show rates for unmoderated studies can be high. You also won't know the quality of the data until you watch the recording. Some sessions may end up being less valuable. Adding more participants to the study can help you avoid these issues.
Finally, one of my favorite tricks is to start a study with moderated usability testing. Once the moderated tests are done, we can pinpoint smaller issues we want to investigate further. I will then run an unmoderated user test on these key points to gather more feedback.
Unmoderated tests aren't for every study, but they can save you time, effort, and money. You can also teach others to set up the tasks, freeing your time to do tests that genuinely require your attention. As much as I disliked unmoderated tests before, they have become a staple methodology and a real lifesaver.
Need to run a large-scale usability test with a tight budget an timeline? Say no more.
With dscout you can...
- Recruit from a pool of quality participants
- Conduct remote research on a lean timeline
- Ensure your research drives organizational impact
And that barely scratches the surface.
See how dscout can not only support your next usability test, but become your all-in-one research tool. Schedule a demo!
Nikki Anderson-Stanier is the founder of User Research Academy and a qualitative researcher with 9 years in the field. She loves solving human problems and petting all the dogs.
To get even more UXR nuggets, check out her user research membership, follow her on LinkedIn, or subscribe to her Substack.
Subscribe To People Nerds
A weekly roundup of interviews, pro tips and original research designed for people who are interested in people
