Skip to content

How to Optimize Unmoderated User Tests

You can't be everywhere at once. That's why unmoderated user testing might be just the thing to scale your research. 

Words by Nikki Anderson-Stanier, Visuals by Thumy Phan

Unmoderated user testing gets a bad reputation. I refused to do unmoderated testing for years and would always preach against it. I would tell others how horrendous unmoderated tests were, what a waste of money they could be, and that the data was virtually useless.

I held a serious grudge.

Now that I'm much further along in my career, I have learned precisely why unmoderated user testing can be so effective—and how to discern when it's the right approach.

Jump to:

What brought me to unmoderated testing

A few years after accepting a position, I was still the only user researcher at a company. Fortunately, my colleagues were super interested in user research. I had research requests coming out of my ears. Some of the asks were for generative research, others for usability tests, and some card sorting. There wasn't enough time in the day, or a week, to get everything done.

I started to include multiple usability tests in one session, and soon I was juggling up to three tests at once. It wasn't realistic, and I felt like I wasn't getting enough information for individual teams. I was a mess. I knew something needed to change when I went into a usability test entirely unprepared. I was as confused as the participants when things did not work as expected, and I forgot the hotspots and flow.

I went back to the drawing board. How could I engage and help all these teams impactfully and still keep my sanity? I had to admit to myself that it was time for unmoderated user testing.

Back to top

What is unmoderated testing?

Unmoderated usability sessions are user tests that the participant completes alone. You build a test similar to a moderated user test, with flows, hot spots, and tasks. However, you then send the link to the participant and have no real-time interaction with them.

The participant then goes through the flow and tasks whenever they have time, and you get sent the recording. With specific tools, you can email questions to the user after the session.

Unmoderated tests are not monitored or guided by a moderator. The participant completes any tasks and answers questions at their own pace, on their own time at location of their choosing.

Back to top

Why you should use unmoderated user testing

As I looked into unmoderated tests more, I saw why others had chosen this method in the past:

✔ Get more responses

You can get many more responses in a shorter time since you are sending a link that multiple users can access at once. Gaining more answers is very helpful for generalizing findings across a larger population. It can also help with teams wary of small numbers that come with qualitative research studies.

✔ Scale up research

They are great for when you're in a rush to get users. If you're juggling many studies requiring you to be a moderator, these tests can get answers while conducting other research.

✔ Collect more data

You collect statistically significant amounts of data that may give you higher confidence in findings. You can segment these tests into different areas (ex: location, age, gender) to see the differences across demographics.

✔ Recruit more participants globally

You may be able to recruit more participants since users can complete sessions whenever they want. You can also access many more types of participants across the world since unmoderated user tests are always remote.

✔ Free up your time

It's easy for other stakeholders to independently set up unmoderated tests (once they learn how to write tasks correctly). By giving them this task, you can free up your time even more.

Back to top

    Usability in action!

    We talked with Design Researcher Meghan Earley about how Dropbox met user needs by going beyond standard usability testing—leveraging video and diary studies to look longitudinally. Check out their field report and see how they were able to gain more confidence in user issue reports and see increased investment from their stakeholders.

    When to use unmoderated user testing

    Like any other user research method, there is a time and a place for unmoderated user testing.

    ✔ Gather feedback on a few specific components or changes

    Unmoderated tests are best for getting input on minor UX or UI changes. Since you cannot explain a complex flow or feature, it's better to use unmoderated user tests for specific elements.

    ✔ Identify small issues on a live website/app

    If you have a screen you want feedback on, unmoderated user tests can be a great friend. Participants can identify minor issues in the UX on a live website or app through specific and straightforward tasks. Keep in mind, you can also use prototypes for unmoderated tests, but they have to be significantly simplified. It's always easier to run unmoderated tests on a live website or app.

    ✔ Discover if users can find relevant product information

    An excellent use for unmoderated tests is to understand if participants can find information on your website/app.

    ✔ Stay on a tight budget

    If you are struggling with budget and also have the right goals, running unmoderated tests can be cheap. Running 25 unmoderated tests can be less expensive than running six moderated usability tests.

    ✔ Obtain a large sample size

    Sometimes teams are wary of large sample sizes, especially if the change can have a substantial financial impact. Just keep in mind the tasks and changes have to be small. However, I have run into situations where a small change could significantly impact the organization. When teams want more confidence in decisions, especially for different segments, unmoderated testing can be a gem.

    ✔ Determine if users understand the value proposition of the product

    You don't have to sequester unmoderated tests to usability feedback. These tests can also give you feedback on your brand and the potential value your organization brings to users.

    ✔ Collect feedback quickly

    We all know about tight deadlines. User research can sometimes get squeezed into the end of a project, causing the timelines to be short. You can run an unmoderated user test over a few days (and the weekend), giving you findings much quicker.

    ✔ Don't expend emotion or imagination

    When I say simple tasks for unmoderated tests, I mean it. Use unmoderated tests for tasks that don't need much emotion or imagination from the participant. The tasks should be simple and easy to follow for the participants.

    Back to top

    When NOT to use unmoderated user testing

    If you are looking to better understand the following situations, I would advise against unmoderated user testing.

    ✘ Complex topics

    If you have a complex topic of testing, unmoderated user tests are not the way to go. You're unable to ask follow-up questions or guide the participant through complicated flows or features. Instead, try moderated usability testing.

    ✘ Early prototypes

    Early-prototype testing is difficult without a moderator to explain and help participants recover from errors or limitations of the prototype. If participants can't understand how to use a prototype, they're likely to drop out or skip tasks, rendering the data unusable. Stick to live websites and apps or extremely simple prototypes that don't require much interaction. For this goal, I recommend moderated usability testing.

    ✘ Deeper understanding

    If you need to understand a particular topic deeply, moderated tests are significantly better. Since there is no chance to ask "why" or dig deeper during unmoderated tests, you'll get more shallow answers. For this, generative sessions (either remote or in-person) would be more helpful.

    ✘ Strong emotional reactions

    Looking for strong emotional reactions from participants is not a great goal for unmoderated user testing. Participants tend to be less engaged without a moderator, so they generally don't emote as much as they would otherwise. If you want strong emotional reactions or imagination, try moderated usability testing.

    Back to top

      How to optimize your unmoderated user tests

      There are some ways to optimize your unmoderated user tests to ensure you get usable and helpful data. Whenever I set up an unmoderated test, I always go through this checklist.

      ✔ Prompt participants to constantly think out loud during the test

      At every screen, remind participants to think out loud. It may seem repetitive and annoying, but you aren't there to nudge them when they go silent.

      ✔ Keep it simple

      The unmoderated test needs to be as simple as possible. Provide enough detail for the participant to complete the task independently and include any information they need to finish the task. For instance, if you need to test a check-out, provide false credit card details for the participant. You want to remove as many blockers as possible.

      ✔ Have many colleagues review the test

      Fine-tune every instruction, task, and question to eliminate the potential for misunderstanding. Participants can easily misunderstand a task or instruction. By getting colleagues to review the tasks, you can check if there is a shared understanding, which increases the likelihood of participants completing the test correctly.

      ✔ Do a dry run see what it's like for the participant

      I always do dry runs before a study begins, even with unmoderated testing. By trying out the tool and experiencing the other side, you can iron out any issues before the participants see the test.

      ✔ Over-recruit participants

      No-show rates for unmoderated studies can be high. You also won't know the quality of the data until you watch the recording. Some sessions may end up being less valuable. Adding more participants to the study can help you avoid these issues.

        Finally, one of my favorite tricks is to start a study with moderated usability testing. Once the moderated tests are done, we can pinpoint smaller issues we want to investigate further. I will then run an unmoderated user test on these key points to gather more feedback.

        Unmoderated tests aren't for every study, but they can save you time, effort, and money. You can also teach others to set up the tasks, freeing your time to do tests that genuinely require your attention. As much as I disliked unmoderated tests before, they have become a staple methodology and a real lifesaver.

        Back to top

        Need to run a large-scale usability test with a tight budget an timeline? Say no more.

        With dscout you can...

        • Recruit from a pool of quality participants
        • Conduct remote research on a lean timeline
        • Ensure your research drives organizational impact

        And that barely scratches the surface.

        See how dscout can not only support your next usability test, but become your all-in-one research tool. Schedule a demo!

        Nikki Anderson-Stanier is the founder of User Research Academy and a qualitative researcher with 9 years in the field. She loves solving human problems and petting all the dogs. 

        To get even more UXR nuggets, check out her user research membershipfollow her on LinkedIn, or subscribe to her Substack.

        Subscribe To People Nerds

        A weekly roundup of interviews, pro tips and original research designed for people who are interested in people

        The Latest