Unmoderated user testing gets a bad reputation. I refused to do unmoderated testing for years and would always preach against it. I would tell others how horrendous unmoderated tests were, what a waste of money they could be, and that the data was virtually useless.
I held a serious grudge.
Fast forward a few years. I was the only user researcher at a company. Fortunately, my colleagues were super interested in user research. I had research requests coming out of my ears. Some of the asks were for generative research, others for usability tests, and some card sorting. There wasn't enough time in the day, or a week, to get everything done.
I started to include multiple usability tests in one session, and soon I was juggling up to three tests at once. It wasn't realistic, and I felt like I wasn't getting enough information for individual teams. I was a mess. I knew something needed to change when I went into a usability test entirely unprepared. I was as confused as the participants when things did not work as expected, and I forgot the hotspots and flow.
I went back to the drawing board. How could I engage and help all these teams impactfully and still keep my sanity? I had to admit to myself that it was time for unmoderated user testing.
What is unmoderated testing?
Unmoderated usability sessions are user tests that the participant completes alone. You build a test similar to a moderated user test, with flows, hot spots, and tasks. However, you then send the link to the participant and have no real-time interaction with them.
The participant then goes through the flow and tasks whenever they have time, and you get sent the recording. With specific tools, you can email questions to the user after the session.
Unmoderated tests are not monitored or guided by a moderator. The participant completes any tasks and answers questions at their own pace, on their own time at location of their choosing.
Why you should use unmoderated user testing
As I looked into unmoderated tests more, I saw why others had chosen this method in the past:
- You can get many more responses in a shorter time since you are sending a link that multiple users can access at once. Gaining more answers is very helpful for generalizing findings across a larger population. It can also help with teams wary of small numbers that come with qualitative research studies.
- They are great for when you are in a rush to get users. If you are juggling many studies requiring you to be a moderator, these tests can get answers while conducting other research.
- You collect statistically significant amounts of data that may give you higher confidence in findings. You can segment these tests into different areas (ex: location, age, gender) to see the differences across demographics.
- You may be able to recruit more participants since users can complete sessions whenever they want.
- You can access many more types of participants across the world since unmoderated user tests are always remote.
- It is easy for other stakeholders to independently set up unmoderated tests (once they learn how to write tasks correctly). By giving them this task, you can free up your time even more.
When to use unmoderated user testing
Like any other user research method, there is a time and a place for unmoderated user testing:
- Gather feedback on a few specific components or changes. Unmoderated tests are best for getting input on minor UX or UI changes. Since you cannot explain a complex flow or feature, it is better to use unmoderated user tests for specific elements.
- Identify small issues on a live website/app. If you have a screen you want feedback on, unmoderated user tests can be a great friend. Participants can identify minor issues in the UX on a live website or app through specific and straightforward tasks. Keep in mind, you can also use prototypes for unmoderated tests, but they have to be significantly simplified. It is always easier to run unmoderated tests on a live website or app.
- Discover if users can find relevant product information. An excellent use for unmoderated tests is to understand if participants can find information on your website/app.
- Tight budget. If you are struggling with budget and also have the right goals, running unmoderated tests can be cheap. Running 25 unmoderated tests can be less expensive than running six moderated usability tests.
- Obtain a large sample size. Sometimes teams are wary of small sample sizes, especially if the change can have a substantial financial impact. Just keep in mind the tasks and changes have to be small. However, I have run into situations where a small change could significantly impact the organization. When teams want more confidence in decisions, especially for different segments, unmoderated testing can be a gem.
- Determine if users understand the value proposition of the product. You don't have to sequester unmoderated tests to usability feedback. These tests can also give you feedback on your brand and the potential value your organization brings to users.
- Collect feedback quickly. We all know about tight deadlines. User research can sometimes get squeezed into the end of a project, causing the timelines to be short. You can run an unmoderated user test over a few days (and the weekend), giving you findings much quicker.
- Not requiring emotion or imagination. When I say simple tasks for unmoderated tests, I mean it. Use unmoderated tests for tasks that don't need much emotion or imagination from the participant. The tasks should be simple and easy to follow for the participants.
When not to use unmoderated user testing
If you are in the following situations, I would advise against unmoderated user testing.
- If you have a complex topic of testing, unmoderated user tests are not the way to go. You are unable to ask follow-up questions or guide the participant through complicated flows or features. Instead, try moderated usability testing.
- Early-prototype testing is difficult without a moderator to explain and help participants recover from errors or limitations of the prototype. If participants can't understand how to use a prototype, they are likely to drop out or skip tasks, rendering the data unusable. Stick to live websites/apps or extremely simple prototypes that don't require much interaction. For this goal, I recommend moderated usability testing.
- If you need to understand a particular topic deeply, moderated tests are significantly better. Since there is no chance to ask "why" or dig deeper during unmoderated tests, you will get more shallow answers. For this, generative sessions (either remote or in-person) would be more helpful.
- Looking for strong emotional reactions from participants is not a great goal for unmoderated user testing. Participants tend to be less engaged without a moderator, so they generally don't emote as much as they would otherwise. If you want strong emotional reactions or imagination, try moderated usability testing.
How to optimize your unmoderated user tests
There are some ways to optimize your unmoderated user tests to ensure you get usable and helpful data. Whenever I set up an unmoderated test, I always go through this checklist.
- Prompt participants to think out loud constantly during the test. At every screen, remind participants to think out loud. It may seem repetitive and annoying, but you aren't there to nudge them when they go silent.
- Keep it simple. The unmoderated test needs to be as simple as possible. Provide enough detail for the participant to complete the task independently and include any information they need to finish the task. For instance, if you need to test a check-out, provide false credit card details for the participant. You want to remove as many blockers as possible.
- Have many colleagues review the test. Fine-tune every instruction, task, and question to eliminate the potential for misunderstanding. Participants can easily misunderstand a task or instruction. By getting colleagues to review the tasks, you can check if there is a shared understanding, which increases the likelihood of participants completing the test correctly.
- Do a dry run see what it's like for the participant. I always do dry runs before a study begins, even with unmoderated testing. By trying out the tool and experiencing the other side, you can iron out any issues before the participants see the test.
- Over recruit participants. No-show rates for unmoderated studies can be high. You also won't know the quality of the data until you watch the recording. Some sessions may end up being less valuable. Adding more participants to the study can help you avoid these issues.
Finally, one of my favorite tricks is to start a study with moderated usability testing. Once the moderated tests are done, we can pinpoint smaller issues we want to investigate further. I will then run an unmoderated user test on these key points to gather more feedback.
Unmoderated tests aren't for every study, but they can save you time, effort, and money. You can also teach others to set up the tasks, freeing your time to do tests that genuinely require your attention. As much as I disliked unmoderated tests, they have become a staple methodology and a real lifesaver.
Nikki Anderson is a qualitative user experience researcher with about 5 years in the field. She loves solving human problems and petting all the dogs. Read more of her work on Medium.