Your Step-by-Step Guide to Setting Up Unmoderated Generative Research

Often, when UX researchers think ‘generative,’ we also think ‘moderated.’ 1:1 interviews and focus groups are common go-to strategies for uncovering more foundational insights about users. We ourselves have even written a step-by-step guide on how to run generative interviews and shared a video with tips.

But that’s not the only way to go about things. Unmoderated studies can also be a fantastic way to uncover foundational insights about your user base. In fact, it offers a few benefits that you don’t always get running interviews.

Jump to:

• Benefits of running unmoderated research
• Step 1: Scoping and goal-setting
• Step 2: Recruitment
• Step 3: Structuring the study
  
• Step 4: Writing the survey
  
• What's next?
  
• Leveraging Dscout for unmoderated generative research

The benefits of running unmoderated research

There are quite a few benefits to letting your research run on its own:

✔ Scalability

Writing an unmoderated study is a flat time cost. It’s a lot of work up-front, but once it’s done, there’s no limit to how many people you send it to. This can be a lot more efficient of a time economy than 1:1 interviews, which scale, well…1:1.

With an unmoderated study, you can increase your sample size with no additional time or effort on your end.

✔ Organization

1:1 interviews require a lot of active note-taking, coding, and/or organization before the data is truly usable. In an unmoderated study, you build structure ahead of time—the data comes out the other side pre-organized and ready to dig into.

✔ Participant Time

When working unmoderated, you have more flexibility in what you ask participants to do than if you’re working within a time-bounded 1:1 interview. It’s more convenient for the participant as well since they can engage on their own time instead of your schedule.

✔ Energy

Frankly, 1:1 interviews can be exhausting, socially, and emotionally. Running an unmoderated study can be a savior not only for your schedule, but for your energy levels as well. They allow you to only communicate with your participants once or twice a day to make sure they are completing the study on schedule.

Reading data on the back end can be taken at your own pace—if you’re studying something more emotionally intense, you can take breaks when it feels good to you.

✔ Breadth

Unmoderated interviews can cover a broad range of topics quickly. This is a good way to get a birds-eye view of your user’s experiences and opinions. If you do want to do moderated research, you can also use unmoderated beforehand to narrow and focus your questioning.

Ultimately ensuring that you’re only asking the questions you already know are going to be fruitful.

✔ Honesty

We all know the issue of interviewer bias in 1:1 interviews. One way to get around this is to simply remove the interviewer. We find that people submitting remote or unmoderated responses on their phones easily forget that someone is on the other end. This means that they may be willing to show and tell more honest information than they might otherwise, depending on the topic.

Of course, it’s easy to say “run unmoderated generative research,” and another thing entirely to actually do it. There’s a lot of planning and choices to make when running these studies—the flexibility can be both exciting and daunting.

One kind of ummoderated study that we talk about often on People Nerds are Diary studies, but that’s not the only way to get exploratory insights out of an unmoderated study.

To walk through how to set up a generative unmoderated study, we put together a real-life example of a simple generative unmoderated survey.

We planned and ran an exploratory survey all about body positivity, body neutrality, and health at every size. We wanted to learn more about these three ideologies and what relationship they had to one another, as well as the role they have played in people’s lives.

We also shared a piece of advice to consider before running sensitive research which refers to this study.

Step 1: Scoping and goal-setting

✔ Choosing a topic

As with any other project, scoping is an incredibly important first step to research success in an unmoderated study. Often, our initial questions are important, but vague. For example, teams have asked us to look into “why users churn.”

Unfortunately, comprehensively tackling this question would translate to projects that are too large to field and analyze in just one study (especially studies with a timeline).

That’s why it’s important to clarify and focus the scope of the project to make sure it’s clear enough to ensure that it's answerable in the time you have allotted, and the answers you’re going to get back are actually what your stakeholders want to know.

This is even more true in unmoderated studies than moderated ones. One of the benefits of moderated generative work, such as 1:1 interviews, is that you have the opportunity to follow interesting threads as they appear.

That means you can sometimes leave the topic of conversation a little broader and drill down on what’s interesting. The improvisational nature of conversation also means you could discover topics you didn’t think of before and find great insights outside of what you thought your scope was.

Unmoderated studies don’t quite have the same luxury. You often only have one shot to ask a specific set of questions, and don’t get the chance to follow up on interesting threads. It’s all the more important to make sure you understand exactly what you want to find out, right at the beginning.

At the end, your scope may have a population to discuss, an experience or topic you want to dive into, and a reason you are interested in this experience.

✔ Questions to ask yourself (or stakeholders) while scoping

• What are your top questions or areas of interest? Which one is the most important?
• Are there any particular experiences that you need to explore? What are some specific “ways in” to your topic?
• What will you do with the information once you have it? What decisions will it be informing?
• Who do you need to ask in order to get the richest answer to this question? Are they readily available for you to talk to?

For example, your stakeholder may say that they want to study “new users.” That’s a good “who,” but not a good “what” or “why.” What about new user experience do they need to find out? Why is it important to them to understand this experience? What are they hoping to affect with this understanding?

After interviewing your stakeholders, you may end up with a more reasonable, answerable scope, which may look more like:

“We want to understand the onboarding process in our platform for new users so that we can understand and decrease roadblocks to getting in our platform and eventually decrease dropoff.”

Or

“We want to understand the triggers for deciding on a product in our space for new and prospective users. This will help us position ourselves better in relation to our competitors.”

When considering our own topic, we originally thought our topic might be ‘body image.’ However, body image has such an enormous scope. What about body image would we talk about? What aspects would we cover? What would we leave out?

Since the concept of body image was way too big for a study, we decided to narrow it down. We asked ourselves:

• What aspects of body image are we interested in learning about?
• What about body image is particularly relevant right now?
• What areas of body image are unfamiliar to us and would bring us insights?
• Who should we ask to get the richest insights while causing the least harm?

With these questions, we landed on the body positivity, body neutrality, and health at every size movements. We also decided that we would study people who are already active in the body positivity or neutrality movements—people who are already comfortable with talking about body image and ideologies.

Additional resources for scoping and goal setting

• Whether or not qualitative research can answer the question
• Generative research: What type of project is this?

✔ Establishing research goals

Once we chose a topic, our next step was to focus even further by determining research goals. Goals are more specific iterations on your general topic.

For us, they act as the benchmark of success—did we achieve these goals, or answer these questions, in the course of our study? Every decision we make comes back to answering our list of research goals, so they’re important to dedicate thought-work to upfront.

To brainstorm goals, we asked similar questions to the above, but about body positivity, body neutrality, and health at every size:

• What do we want to learn about body positivity, body neutrality, and health at every size?
• What type of experiences do we want to learn about?
• What information do we want at the end of the study?

With these questions, we crafted the following goals:

• Discover the paths people take to get into the body positivity, body neutrality, or health at every size movement
• Learn about people's philosophies of body positivity, body neutrality, and health at every size
• Uncover how people interact with or experience body positivity, body neutrality, and health at every size in their day-to-day lives
• Discover resources people interact with to gain inspiration from body positivity, body neutrality, and health at every size

✔ Methods

Lastly, we discussed methods for achieving these goals. We knew we needed to get in-depth information from our participants to answer our questions. But we decided that we could get much of what we needed from an unmoderated study on Dscout. We also chose this method in consideration of the sensitive topic at hand.

We decided to gather information using Dscout’s Diary tool and then clarify or learn more through a few 1:1 interviews after the initial study using Dscout Live. We used this information to scope a project timeline with Miro.

Once we had our topic, goals and methods established, we were ready to write a research plan.

Step 2: Recruitment

✔ How to find the right people

Once you have your research plan drafted out, it’s time to find your participants.

Before you begin, it may be helpful to conduct some desk research to understand how to reach the right people (and what to ask them once you find them). We didn’t know too much about body positivity and surrounding movements, so we conducted some desk research to make sure we asked knowledgeable and sensitive questions.

Now that we had a basic outline of our goals and the methodology, it was time to think about who we would recruit. Since we didn't know too much about the topic, we conducted to ensure we spoke correctly about the different topics. We mainly used two approaches to desk research:

1. Reading up about and better understanding the body positivity, body neutrality, and health at every size movements from verified sources
2. Speaking with colleagues and others who were part of these movements and could look over our questions (thanks to registered dietitian Stephanie Dorfman and leadership coach Jocelyn Resnick)

✔ Screening

We had to determine the participants' criteria we were looking for. These criteria were directly related to being able to answer the research goals. We wanted to ensure we recruited participants who could help us answer and achieve our research goals.

We came up with the following criteria:

1. Those who are directly involved in the body positivity, body neutrality, or health at every size movement through a variety of ways
2. People who have heard about and are knowledgeable able about the body positivity, body neutrality, or health at every size movement
3. Those involved in nonprofits or within a career inside these movements
4. People impacted by body positivity, body neutrality, or health at every size
5. Those who practice body positivity, body neutrality, or health at every size

We also needed to make a decision on how many participants we wanted for the study. Ideally, since unmoderated studies like this are a kind of qualitative research, we wanted at least fifteen. However, in the end, we settled for having at least twenty participants. We recruited 25 to account for potential attrition during the study.

We used the Dscout platform to create a screener survey. We decided on a mix of closed and open-ended questions because some of the questions were too hard to make into closed questions, and we wanted to see how the participants responded.

It is helpful to have this mix of closed and open-ended questions in a study where you need people to submit information or go into detail on your topic willingly. Through the video screener and the open-ended questions, we got a much better idea of how people might participate in the study and the level of detail we could expect.

✔ Screener design

Screener teaser:

Are you involved and passionate about the body positivity, body neutrality, or health at every size (HAEs) movement? Now is your chance to share your experience with this movement, your journey, how it has impacted your life, and advice to others! Apply now for a chance to participate!

If selected for this project, you will share your experiences with us in a series of six parts over ten days and might be invited for a live interview to follow up. Scouts who complete the project will earn $110!

Screener questions

After sending the survey out, we capped the number of responses at 500. In the end, we ended up with 487 applicants, which was quite a lot to go through! Luckily, the Dscout platform automatically flags "low-quality" participants and some "bad fit" participants.

So we went through the participants, highlighting those who could be a "good fit" or a "possible fit." We categorized the participants based on the research goals and the criteria we determined at the beginning of the study. We kept those criteria up as we went through each participant to keep who we were looking for top of mind.

We ended up with 32 "good fits" and 27 "possible fits." We wanted to over-index on the "good fits" to have additional backups ready if people didn't accept our invitation.

In addition to labeling the fit of each respondent, we also took the "good fits" and broke them up into an "initial" group and a "backup" group. This grouping made it easier for us to know who we were sending the first invitations to and who we would send the next wave to if we didn't get enough responses.

Step 3: Structuring the study

✔ Unmoderated drafting overview

Just like an interview guide, you can (should) break up your unmoderated survey into chunks that are easy for your participants to get through. Each part should focus on one activity or topic, just like a subsection in an interview guide.

More often than not, your research goals are a great place to start when figuring out what parts should be included in your mission. We decided on a 6-part structure, each of which helped to achieve a particular goal, as seen below:

Each part we draft has its own question flow and natural end point, so participants can dig deep into each individual topic without getting overwhelmed—just as if we were moderating live.

✔ Details and logistics

With a diary study comes detail and logistics. Before letting your diary study out into the wild, there is a lot to think about. Here is what we had to consider:

• Length of the study
• Part availability (automatic or manual)
• Incentive amount
• The overview
• Device

✔ Length

Participants need time to fit in research to their busy schedules. It’s generally best to give participants at least 1-2 days for a simple research activity. If the activity or topic is more complex (e.g. needs multiple submissions, or for them to complete a task prior to finishing the activity), tacking on a little extra time is thoughtful.

Our study had six parts with varying levels of complexity. We scheduled out one day for some simpler parts, 2-3 for more complex ones, which eventually gave us a total of ten days.

✔ Part availability

On Dscout, you can choose two different ways to schedule your diary study:

• Manual: In this mode, you control when parts become available to scouts. The first part will open to scouts on launch. Subsequent parts will be in draft mode, and you can edit them until they are published and launched to scouts. Note that parts must be opened sequentially.
• Automatic: Automatic missions give scouts more control and require less work on the researcher's part. The first part will open to scouts on launch, and all subsequent parts will be locked from being edited. Once a scout submits the minimum number of entries required for a part, the next part automatically opens up for them.

For our study, we decided on an automatic mission. We wanted our scouts to have the flexibility and convenience of answering questions at whatever pace felt comfortable over the 10-day fielding.

✔ Device

It’s also worth considering what device participants have available to them. Will they be typing on a keyboard, talking, or typing with their thumbs? Will they be able to take their device into the context you want?

On Dscout, a researcher can choose whether a participant can fill out their study on mobile (portable and good for app-based research, but difficult to type longer responses) or on desktop (good for longer responses and web-based research, but can’t be moved around easily).

Because this mission had a lot of thoughtful questions that might require a lot of typing, we asked participants to fill out most of their parts on desktop. But for the parts where they might want to show us an app on their phone, we opted to do mobile.

✔ Incentive

Deciding how much a survey is worth can be an amorphous and difficult task. A good rule of thumb is to pay at least $10 per research activity (or per entry, if you’re on Dscout) and add additional compensation on for any added complexity, time cost, or emotional cost that your study may include.

Since the diary study is remote, automatic, and none of the parts required too much time, we required a total of 9 entries over 6 parts, which is a fair amount!

We also recognized that talking about body image requires a lot of thought work and potentially some heavy emotional lifting which we also wanted to compensate for. We landed on $110 per participant. This incentive excluded any one-on-one interview we would ask select participants to.

Step 4: Writing the Survey

Now for the nuts and bolts—the actual writing of the qualitative unmoderated study.

As we’ve said before, writing a survey instead of an interview means that you can’t iterate, dig deeper, or clarify your meaning to your participants during the study. Lots of that thought work has to be done ahead of time.

But with good preparation and well-worded questions, you’ll end up with great data that’s a lot easier to sort through and reference than a full interview. Here are a few pro-tips on how to get to high-quality data using survey questions:

✔ Consider how your question could be misinterpreted

You won’t have a chance to clarify your language to your participants when your survey is live, so spend some extra time before it launches considering each way your question could be misinterpreted.

I personally think this process is kind of fun: try and answer your questions in the silliest, most bad-faith way possible, and then put up guard rails in the question to avoid that interpretation. Don’t feel afraid to explain yourself by using phrases like “when we say the word ____, we mean _____”.

✔ Avoid double-barrelling

Don’t add more than one question in a given prompt—it will give participants the opportunity to only answer half of what you ask. It’s always better to have multiple granular questions than one large question.

For example, don’t ask “how do you feel about this topic and why?” Instead, try two questions:

“How do you feel about this topic?” followed by “Tell us more about that. What experiences have led you to feel this way about this topic?”

✔ Specify what you would like your participants’ responses to look like

Participants can’t read your mind! Give a sense of how much you’d like them to say, and in what format, so they can get a sense of how in-depth you want them to do. Phrases like “In three words”, “In a few sentences,” “Give a brief list,” or “Be as detailed as possible” are your friends.

✔ Use close-ended prompts to set up your questions

Often, we’re qualitatively interested in the “whys” of the world instead of the “whats.” One way to cut to the chase is to use a close-ended question to get at the “what” answer, and then use an open-ended follow-up question to get the reasoning behind it.

For example, instead of asking “How happy do you feel today and why?” (Doubly ineffective, because it’s a double-barrelled question!) Instead try a close-ended question like “On a scale of 1-5, how do you rate your mood today?” and follow it up with “Why did you rate your mood the way you did?”

✔ Lean on video for multi-facted or emotional questions

Survey questions are generally text, but in Dscout, we can collect one video per part. This makes video data a precious resource that we can spend on particularly thorny questions.

Try to use your video question on questions that are too multi-faceted to break up into open ends, or on questions that would really benefit from seeing the participant’s reaction as they tell you about their answer.

These are some of the top points we kept in mind while we were writing our own survey questions. We’ll show you two examples of parts we drafted below, along with a few pointers we considered along the way.

Part 1: Your Philosophy

In this section, we wanted to ask questions that helped us understand how each participant defined body positivity, body neutrality, or health at every size.

This part was relatively straightforward, and only has 1 entry (an entry, as we’re using it here, means a participant’s response to a set of questions. If a research activity is completed multiple times, that activity would have multiple entries.) that scouts must submit before they are done. Below you can see the logistics of the section:

✔ Instructions

In this part, we want to understand your philosophy of body positivity, body neutrality, and/or health at every size. We want to hear how you define these concepts yourself and explain them to others.

There is one entry to complete in this part. Once you complete it, Part two will automatically unlock for you.

Please note that most questions in this part are optional. Remember that answers you give may be published anonymously on our public blog. If you ever feel uncomfortable answering a particular question, you don't have to! If you wish not to answer, just say so. Or, just use the code-word pineapple ? in place of an answer.

Number of entries: 1

Number of days: 2

We estimated that it shouldn't take more than two days to complete this part. This also helped us ensure that people completed this part within the two days to identify any drop-outs early on.

Part design:

This is what we call a “headline question.” A headline question helps you keep a title for a particular part of your study.

In Dscout, this is literal—a question you mark as a headline will appear as the ‘title’ of the entry across the platform, making it easier to skim and organize. But even if your platform doesn’t accommodate this, a ‘headline’ can still be a useful memory tool when you’re sorting through data down the road.

We did this as we had three concepts and wanted to understand which one was the focus in this part. This question also gave us a great idea of how many people talked about body positivity versus neutrality versus health at every size.

We used the video question for the meatiest and most dense question in the part, because people often process and communicate more while talking than typing.

We asked for "up to three" drawbacks and benefits of the movement they chose. This ensured that we got enough information from the participant (and not just one-worded answers), but not too much that we would be inundated with qualitative data. We used this method on many of our open-ended questions so that participants had a better idea of what we were expecting.

The question we’re interested in is really question 6, not question 5. But we asked a close-ended question first to help organize the responses to question 6, and also get people thinking about the topic more concretely before jumping into an abstract open-ended question.

Part 2: Impact of Technology

This part was slightly different from part one. For this part, we wanted participants to show us several different instances of how technology impacts these movements. This means that scouts had to submit information to the part multiple times, adding to the complexity of the part.

✔ Instructions

Welcome to part two! Now that you told us about your philosophy, we want to understand how you think technology impacts body positivity, neutrality, and/or health at every size.

In this part, you will fill out three entries.

- In the first entry, you'll tell us your overall opinion on how technology influences body positivity, neutrality, or HAES movements.

- In the second entry, you'll show us one example of how technology impacts these movements positively.

- In the final entry, you'll show us one example of how technology impacts these movements negatively.

Complete all three entries in this part to automatically unlock Part Three.

Please note that most questions in this part are optional. Remember that the answers you give may be published anonymously on our public blog. If you ever feel uncomfortable answering a particular question, you don't have to! If you wish not to answer, just say so. Or, just use the code-word pineapple ? in place of an answer.

Number of entries: 3

We had participants fill out this part three times, as described in the instructions above. These could have been three separate parts as well, but we thought that because the topics were so similar thematically, it would make more sense as a single ‘activity’ for them to complete.

This method was a great way to capture multiple perspectives from the participant about a single topic.

Number of days: 2

Although this part is slightly more complicated than a single-entry part, we decided that it was still short enough that 2 days would be enough to complete it.

Part design:

Again, we started with a headline question. This time, though, it also had skip logic attached to it: not only would it help us organize our thoughts, but it would also pipe participants to particular messaging following the question.

Track 1: General thoughts on technology

If scouts selected that they were filling out information on general impact of technology, they went straight on to the following question track.

We made sure to split this into two questions, to make sure that scouts are never thinking about answering more than one specific question at a time. In a 1:1 interview, we might be able to get away with asking, “What are the positive and negative effects of tech on body positivity?” But not here!

We started with asking pros and cons so that our participants could think through each, and then asked to weigh them and see which came out on top.

Once again, we set up our open-ended response (Q4) with a close-ended question (Q3).This can be particulary useful when scouts are filming a video. That way they don’t waste time in the video itself thinking through the thesis of their opinion - they already have it in mind and they’re ready to dig into the more interesting “why” behind their answer.

Track 2: Examples of positive and negative moments

If scouts selected one of the second two answers to the first question, we skipped them down to a different question set. For example, let’s imagine they selected “a negative example of technology”, here’s what they would see:

This is a checkpoint—no answer required. It’s just some instructions to set scouts up for their task so they’re not taken off guard as they continue through the part.

As you can tell by now, we love a good headline question. This one wasn’t an official ‘headline’ in our platform, but was still very useful when we were skimming through examples looking for particular apps or themes.

We decided to have participants screen record these answers to show us tangible examples and explain why they thought it was a negative or positive impact example. Diary studies are great for seeing instead of just having participants tell you what happened. Utilizing screen recording and videos was a great way to see participants in their element and get rich data.

Note here that we asked multiple questions at once (what is it? Where did you find it? Why do you think it’s negative?) This would otherwise be multiple written open-ended questions, but because they’re talking for a longer period of time, they have the opportunity to move through multiple prompts without missing any. That being said, the more questions you add to your video, the more you can expect scouts to forget to answer one or more of them or run out of time before they get to it.

This question is invaluable, regardless of what you’re studying. You might not get immediately actionable answers, but you will always get new insight into the real unmet needs of your participants.

What’s next?

In a moderated setting, this is right when the real work would begin: back-to-back observation periods, 1:1 interviews, focus groups, or any number of other methods that would eat days or weeks of our research time.

But that’s the best part of unmoderated research—once you’ve set up, you’re practically done.

Your next step will be to field your study. This means sending it to participants, and waiting patiently while they return your results. Our study was in our participants’ hands for about two weeks, during which we spent about 30-60 minutes a day checking in on their progress, answering questions, and sending encouragement. At the end of the time period, we had expended almost no time or energy and had an absolute mountain of data to work with.

Another fantastic element of unmoderated research is how ready the data is to analyze. In moderated research, there’s a lot of work to do to make the data digestible: some people take comprehensive notes, some code their data extensively, some write interview summaries and still others hand-transcribe key quotes. All this before the really meaty thematic analysis!

With unmoderated research, much of that pre-work is done for you. Your data is already in recorded bite-sized chunks organized by subject, since you asked such specific open-ended questions. Spoken data is also pre-transcribed and in small pieces, ready to be tagged or put into playlists. And your close-ended questions act much like a comprehensive coding would in a less structured study: they’re ready-made tools that you can organize and compare your data by.

Now all that’s left is to dig in and pull out your new insights. We know, easier said than done! Fortunately, People Nerds has already written about a variety of ways you might approach your analysis phase with an unmoderated study:

• Foolproof Qualitative Analysis Tactics
• Avoid getting stuck in user research analysis
• How to Code and Tag Qualitative Data
• Using Dscout’s built-in analysis tools to spot trends
• Cracking the Qualitative Research Code
• Three Approaches to Tagging that bring Clarity to Qual Data

With the right setup, the insights your data produce will be just as rich as any moderated generative work you could do, with a fraction of the fielding time.

Interested in learning how Dscout can help you conduct impactful unmoderated generative research?

Let's connect!