Set the Table for These Multi-Course Concept Tests
Combine dscout tools for a smooth broad-then-deep evaluative research design.
Words by Emily Quandt, Visuals by Jarred Kolar
Evaluative research—like sourcing feedback on in-flight or pre-production concepts—is a primary research goal for UXRs, designers, and product folks. Choosing between paths often requires MORE than simply a usability test or a few interviews...a confident decision would combine these approaches, maximizing the diversity of data.
And there are lots of questions surrounding a concept test:
In what format should I present my concepts?
How do I want participants to access it?
Should I focus on breadth or depth?
Do I want to prompt participants or do I want their free thoughts?
This recipe moves those questions from "either/or" to "yes/and." It's flexible, molds to your team's specific needs, and can be used with a variety of participants and timelines. Flexing between Diary and Express offers the choice between longer, deeper work, and quick-turn.
So whether you have two days or two weeks, this recipe will offer a breadth-and-depth design to answer both your generative (how might someone use this) and evaluative (which do they prefer) questions. The use of different dscout tools also offers a rinse-and-repeat approach if your design team comes back with a different direction; you won't need to recruit over and over.
There are a few key decisions you’ll want to make upfront to ensure a smooth research project and end results that will be relevant, useful, and impactful.
First, choose your course of action. Do you want participants to see your concept(s) and be prompted to answer specific reaction questions? Do you want to capture natural moments over a period of time, and then show your concept(s)? It's helpful to keep in mind the different strengths of each of dscout's three tools: Diary, Express, and Live.
Depending on your ideal course of action, here are some course combinations for inspiration:
A moments-based Diary mission (to capture current/ideal use cases) +
A Live session to show your concept(s) and capture reactions
A Live session to show your concept(s) +
A moments-based Diary mission to surface potential use cases
An Express mission to capture current use cases +
A moments-based Diary mission to capture ideal use cases +
A Live session to unpack user reactions to your concept(s)
In this recipe, we’ll outline a two-course meal—i.e. a two-mission concept test—starting with a Diary or Express mission followed by a Live session. Feel free to mix this up, fitting it to your own research needs! Remember: Diary and Express are unmoderated tools, while Live is moderated. Adjust your needs accordingly.
Second, decide the type and scale of data you’d like to collect. Are you hoping to create charts or graphs to show users’ reactions? Do you want a few supporting quotes? Knowing your ideal outcome will help you ask the right questions and prevent data overload.
Finally, consider whether you want current users, prospective users, the general public, or all of the above! To make your decisions, ask yourself what your end goal is: is the purpose of your project to discover what your users want? Is it to uncover what might lead to greater adoption? Is it to understand what the general population wants or doesn’t want? Once you have an idea, you’re ready for Step 2.
Step 2: Recruit
You can either bring your own participants or use the dscout platform to do your recruiting.
Skip this step if you’d like to use an Express mission (see the next step).
Step 3: Run your first mission
Depending on the depth of your research, you can use a Diary or Express mission to ask your closed-ended questions and (if you choose) show your concept. If you're on the fence about which tool to choose, here is a quick comparison and potential impacts on questions:
Express missions are best suited for fast research or getting gut reactions from participants without going into too much detail. They’re also more affordable, with incentives between $.50 to $5. If you have more than 1-3 concepts to share, we recommend using a Diary mission.
Use dscout Express to ask a few qualifying questions to narrow your pool down to the user-base you’d like to hear from. Then, show your concept and add scale- or ranking-style questions to collect reactions. Add a single open-ended question or 30-second video prompt to ask why they rated it that way.
Express sample outline
Start with 2-4 qualifying questions with knockout or must-select logic.
Add a checkpoint, closed-ended, or open-ended question to attach your stim (the concept[s] you’d like to share).
Add 2-3 ranking or scale questions.
Add 1 follow-up open-ended or video question.
Diary missions are best suited for more in-depth research, giving participants more than one day to complete. Diary missions are also more suitable if you plan to ask multiple open-ended questions.
When using a Diary mission, you won’t need to ask qualifying questions; instead, jump straight into your question set.
Diary sample outline
Part 1: General thoughts
- Gauge opinions on the topic with a few scale or ranking questions
- Add a video prompt to explain. (Tip: you can also use a video or screen recording prompt to see how participants currently use a product or service.)
Part 2: Wishes
- Ask participants what’s going well in an open-ended question.
- Have participants draw their ideal solution with a photo prompt.
Part 3: Concept reactions
- Show your concept(s) by attaching as stim or linking to a file or website.
- Add 2-3 ranking or scale questions.
- Add 1 follow-up open-ended or video question. (Note: You may forego this part and wait to show concepts during a Live interview session.)
Step 4: Review entries to select Live session participants
Whether you choose Diary or Express, you’ll have lots of entries to review! Use these entries to choose who you’d like to invite to your Live mission.
There are two main factors we recommend considering:
Select scouts who are expressive and detailed in their responses to ensure they’re engaged in the research, optimizing your chance for a great interview. Additionally, be sure to select a variety of scouts with differing ratings of the concepts to hear from all sides.
Step 5: Conduct Live interviews
If you’ve already shown your concept(s) to participants, you may use this time to gain further insight into their reactions. Otherwise, you may use a Live interview to show your concept(s) and get reactions recorded in real-time. Either way, be sure to have a discussion guide ready to go—you can even practice fielding your Live mission by using the test mode.
Sharing the insights is just as flexible as the recipe itself. You could simply share the "winner" based on participant scores, however there are "extra mile" options to consider that can all be created using features native to dscout:
Create a clip playlist from video collected during interviews or missions
Offer collaborators access to mission or session data to poke around themself
Word clouds of sessions might be used to highlight pain points or delight moments
Emily Quandt is a Research Advisor at dscout with a background in human ecology and research in the public, private, and government sectors. When she isn't heads down in a project, you can likely find her in the woods or on the water with a dog companion.
Subscribe To People Nerds
A weekly roundup of interviews, pro tips and original research designed for people who are interested in people