Sample Study Designs: Concept Testing Three Ways
Concept testing gives your team a sense of direction, prior to the development of a prototype. Here are three approaches for impactful concept testing with dscout.
Sometimes you feel “between” discovery research and usability testing. Your team has a few concepts in mind, but they aren't fully formed ideas. They want to test to see if they are going in the right direction, but the concepts aren't usable.
Concept testing encourages you to put early-stage ideas in front of participants to determine which direction the team should go in or whether or not your team is going in the right direction. These tests will help you understand how people feel about your concepts/ideas.
Here are three sample dscout mission designs for concept testing in dscout—plus some upfront advice on when to use each one.
First: Decide on a method
Concept testing can work in a variety of ways. Choose the one that’s best for your bandwidth and for what your team needs.
Moderated concept testing using Live
This lets you get in-depth feedback and insights about one or two concepts. You can understand how people are interpreting and interacting with the concept, get detailed design feedback, and dig deep into emerging areas of interest. Running Live interviews are rewarding, but time- and resource-intensive.
Unmoderated testing using Diary
Gets you a bigger sample size than Live, and requires less resources. It also allows you to test a wider range of concepts without eating up too much of the scouts’ or researchers’ time. Diary is also good for comparing multiple concepts against each other with both qualitative and quantitative metrics.
Unmoderated testing using Express
If you are looking for quick, broad directional insights, Express will allow you to ask a brief series of questions about your concept(s) and get you feedback in record time, all with minimal effort on your part.
Mixed methods
An Express mission is a great first stop along the way to a bigger study. Use a quick-hit survey to help crystallize which concepts warrant further examination, either with a Diary or a Live mission.
Next: Decide how you’re going to present your concepts
Another big decision worth making before you design your study: How will you communicate your concept(s) to the users? Our teams at dscout have done this in a variety of ways:
Moderated: share your screen
Perhaps the most straightforward way to share concepts. Use Live to interview scouts, and ask them to respond to
Moderated stim
You also have the option to add stim to your Live missions, which will give scout some control, which may be useful if there’s a lot for them to explore, or if it’s an interactive wireframe or prototype that you want them to look through.
Unmoderated stim
Stim in the unmoderated/Diary dscout tool allows you to upload still images to survey questions. This is a good option if your concept is relatively simple to get across, or if it can mostly be described in text with some visual aid. Program in a question with a text description of your concept and use stim to illustrate your idea.
An external website
Create a private micro-site with separate pages for each concept, including images, descriptions, use cases, or anything else you want scouts to respond to. You can include relevant links in your dscout study, or you can send Live participants a link ahead of time if you prefer for them to spend time reviewing the concept before your call.
Method 1: Moderated concept testing
Moderated concept testing is a good way to get detailed, in-depth feedback on your concepts. It can be time- and work-intensive, but will be worth it for the level of fidelity you will get on your samples.
Recruit and sample size
For in-depth concept feedback, you will likely want people who are already knowledgeable about your product (or about similar products) and who are going to be articulate.
Your sample size doesn’t need to be large for this study, and will be in part determined by how much bandwidth you have for interviewing. We suggest 5-10 people per key segment you want to look at.
Method 1: Example design
| Introduction: 3 minutes about the test, who I am, signing an NDA/consent form, instructions |
| Warm-up: 5 minutes for asking general questions to get the participant in the mindset of conversation. My favorite warm-up questions are, "what hobbies do you love?" "what do you do in your free time?" or "what have you watched recently that you loved?" |
| General questions: 10 minutes to focus on the problem-space of the concepts. For example, if we were testing different meal kit plans, we would use this section to ask about meal kits or cooking habits, in general. |
| Concept A: 20 minutes focusing on concept A. I start with general impressions and have participants explain the concept to me, as they would friends or family members. Then, I dive deeper into the different details of the concept. |
| Concept B: 20 minutes focusing on concept B in the same way as above. |
| Follow-up (optional): A 5-minute buffer to follow up on the concepts and check if the participant has anything else to talk about or add. |
| Outro: 2 minutes of thanking the participant, answering any of their questions, and explaining any next steps, such as when they can expect the incentive. |
Method 2: Unmoderated concept comparison
Unmoderated concept testing has the advantage of reaching a lot of scouts with less time and effort. Your feedback won’t be quite as detailed as it is in moderated testing, but will still give you a broad sense of what people are responding to in concepts.
Designing an unmoderated concept study also allows you more space to ask foundational questions relating to your concepts. This is optional, and your team may have already done this work. But survey-style testing will allow you to kill two birds with one stone.
Recruit and sample size
Who you recruit will depend on the segments you are trying to reach with your new concepts. Consider:
- Power customers who know a lot about your product
- Light customers (the concepts might get them to increase their usage)
- Prospective customers (the concepts might get them to switch to your product)
A mix may also be beneficial. Also consider any key segments your company wants to examine.
Sample size can vary depending on your bandwidth. The sample can be as small as a moderated study (~5-10 scouts per segment), but can also be much larger without much additional effort on your part. Making your study automatic will decrease on scout management needs, making this an even lighter lift.
A larger study (25+ scouts per segment) can have the added benefit of allowing some quantitative reporting (though these numbers will still be representative or definitive).
Part 1: Understand the current situation
This part serves multiple purposes. First, it lets you get to know your scouts and gives important context to later answers. It also warms them up to the space you’re going to be asking about, which can be crucial since concept testing can be a new cognitive space for people. And most importantly, it gives you a baseline of how users are thinking about your product or product space, and what issues exist around it.
| Example questions: |
| How often do you do [activity related to concept]? |
| What are the biggest challenges in [activity related to concept]? |
| How do you currently get around those challenges? |
| If you could wish for a new solution to those challenges, what would it be? |
Part 1 can just be a quick introduction; but if you desire more information, additional questions or even additional parts can be added to increase the detail you get at this foundational stage. For example, consider asking scouts to:
- Record a video telling you about their biggest barriers to success are in your problem space
- Inventory all the current tools and solutions they have to whatever problem you’re trying to solve for
- Sketch out their own concepts for solutions to their problems
- Compare your product with competitors on how well it currently solves their problems
- Capture in-the-moment activities related to your concept
Part 2(+): Testing concepts
This is the real meat of the design. Dedicate one part to each individual concept you would like to test. Each part should be identical to one another for comparative purposes.
In these parts, close-ended questions will be very useful: they will let you quickly compare impressions of your various concepts on the close-ended response analysis page, and will also let you filter to subsets of opinions that you want to read more about.
In the introduction and/or first question of each part, include the concept that you wish to get feedback on. This can be a checkpoint with a description, added stimulus for a visual concept, or a link to a private website for a more in-depth concept description.
| Example questions: |
| How intrigued/compelled are you by this concept? Why? |
| How applicable do you think this concept is in your life? Why? |
| How would this concept improve (and/or not improve) the experience of [activity related to concept]? |
| How would you use this concept in your daily life? |
| If this concept were a reality, do you think you would engage in [activity related to concept] more or less than you usually do? Why? |
| Do you think this concept is unique? If not, what does it remind you of? |
| What stands out to you as great about this concept? |
| What stands out to you as concerning about this concept? |
Final part: Reflect and compare
After you’ve finished running through all your concepts, give users a moment to reflect and review all your concepts as a whole. Find out which one was most appealing to each scout, and take the opportunity to collect some verbatim quotes that will speak directly to each in comparison.
| Example questions: |
| Review all the concepts we’ve presented to you and rank them in order of most interesting to least interesting. |
| Review all the concepts we’ve presented to you and rank them in order of how often you would use them. |
| What about the top ranked concept do you like the most? What makes it better than the others? |
| What could change about it to make it even better? |
| What about the bottom ranked concept do you like the least? What makes it worse than the others? |
| What would need to change about the bottom ranked concept to make it more appealing? |
| What activity/use case do you think would benefit MOST from one of these concepts? Why? |
| Any last thoughts about any of these concepts? |
Method 3: Quick-hit concept impressions
If you’re looking for quick and dirty feedback on a couple of concepts to see if you’re on the right track with your team, then an Express mission may be all you need. To do this, condense each part in the suggested diary design to 1-3 questions and launch a single survey that captures feedback for all your concepts on a high-level.
Method 3: Example design
| How often do you currently do [activity related to X]? |
| How enjoyable is this activity currently? |
| What could change to make this activity better? |
| Please look at concept 1. Overall, how compelling is this concept? (scale question) |
| Explain why it is compelling or not compelling. |
| How applicable is this concept? (scale question) |
| What use do you think this concept would have in your life? |
| What are 3 words you would use to describe this concept? |
| [repeat Q4-Q8 for subsequent concepts] |
| Rank all the concepts you’ve seen. |
| Which did you rank #1 and why? What makes it better than the rest? |
| Which did you rank last and why? What makes it worse than the rest? |
Karen is a researcher at dscout. She has a master’s degree in linguistics and loves learning about how people communicate with each other. Her specialty is in gender representation in children’s media, and she’ll talk your ear off about Disney Princesses if given half the chance.
