March 6, 2024
March 6, 2024
Qualitative research can be incredibly fun and exciting. You’re able to walk into sessions and walk out with a new understanding of participants' lives.
It’s fascinating to understand what and how people think about different concepts. From travel to hospitality and social media to body image, I never get tired of talking to people about how various topics impact their lives.
When I first started conducting generative research, I was in a state of bliss. I couldn't believe that doing something so enjoyable was my job and something I got paid for. Yes, synthesis was complex, but I managed to get through it and even included debriefs with stakeholders in my process.
After 10-20 interviews, I had mounds of data after synthesis. How was I supposed to sort through all of this information and present the “most important” information?
Plus, importance is subjective and not one participant told me what was most important to them when describing painful experiences or goals they needed to achieve.
I assigned the importance to insights and presented this to my team for a while, but it felt disingenuous. And, when asked seemingly simple questions like, "How do we know this is what we should focus on?" I struggled to answer. I also couldn't argue with that old, dreaded phrase of, " But we only spoke to fifteen people."
I set out to fix this recurring issue. I wanted my insights to be valid and reliable, and I wanted to be confident when presenting to my teams.
I knew that what I wanted to do was generalize my qualitative insights across a larger population. Yes, I was speaking to the right sample size of people for my method, but I wanted to be sure that these insights could be valid and reliable across the entire segment.
With that as my goal, I knew I had to dip into my previous training in quantitative methods. Surveys would allow me to reach a more extensive population and understand if the broader audience aligned with my insights.
This approach would give me that level of confidence when I was presenting to colleagues. I could assuredly say what we needed to focus on next and give my teams the direction they asked for.
After some research, I decided to try two different methods of surveys: a feature request survey and the opportunity gap survey.
The feature request survey was my first iteration on applying my insights to a wider population. I still use this approach today, however, less than the opportunity gap survey. I find it efficient and effective, especially when I have to get a prioritized list of features to my teams quickly.
This survey is most helpful when your team asks the question, "What should we do next?" or "What features do users need the most?"
When teams come to me with these questions, I conduct a quick generative research study if I don't already have enough research to put together a list. In this study, I aim to learn about the goals users are trying to achieve on our platform and the main pain points they are encountering.
Let's dive into an example about a weather forecasting app (because, hey, maybe we can leave our houses a bit more now).
Say my team approached me asking what they should build next, based on the user's needs. In this study, I would focus on understanding:
After conducting the research and synthesizing, I have clusters of information. The larger the groups meaning more participants have felt/thought that particular sentiment, are considered the top themes.
Let's say the major themes are:
Armed with this knowledge, I build a feature request survey by turning these into ideas. First, I use the user story format to convert them and then take all the extra wording out for the survey.
These would look like:
After writing through these user stories, I would set up the feature request survey. In this survey, I ask users to rank the features they would like to see, and which are most important to them.
Please rank the level of importance of these features based on your day-to-day needs• Knowing the weather for the exact hours you will be outside
• Seeing yesterday's temperature in addition to the weekly temperatures
• Notifications that alert you to significant upcoming weather changes
• Suggestions on how to dress or what weather accessories you should bring for the day
• A quick scan of the day's weather and what you should be prepared for
• Information on the road conditions and what to expect
• A personal forecast based on your schedule and commutes
• Information on the different weather terms (such as 50% rain) and what they mean
I would send this survey to the appropriate sample size and use frequency to indicate what we should focus on next, resulting in a chart that could visualize the most highly ranked feature ideas.
Something to keep in mind with feature requests surveys is that they are technically future-based and asking people what they want. These two concepts can be tricky to capture through user research, which is why the opportunity gap survey may serve you better.
The opportunity gap survey is part of the jobs-to-be-done framework. However, I’ve also used it for other generative research sessions.
This survey analyzes the gap between how important each insight is and how satisfied people are with current solutions. These results give us more context than the above feature request survey, but it does take longer. We can use this information to compute the opportunity score of each insight.
Let's take the example from above and turn that into an opportunity gap survey. First, you need to conduct research to figure out the insights, and then you ask each survey respondent two questions per insight:
1. Insight: Knowing the weather for the exact hours you will be outside
• How important is it for you to [know the weather for the exact hours you will be outside]?
• How satisfied are you with how you can currently [know the weather for the exact hours you will be outside]?2. Insight: Suggestions on how to dress or what weather accessories you should bring for the day
• How important is it for you to [know how to dress or what weather accessories you should bring for the day]?
• How satisfied are you with how you can currently [how to dress or what weather accessories you should bring for the day]?
Once you receive your responses, you can calculate the opportunity score or plot the answers on a graph like below:
The work you should prioritize are insights with a low level of satisfaction and a high level of importance. Afterwards you can focus on those with a high level of importance and a mid-level of satisfaction (ex: importance three or above and satisfaction at around three). You can use this type of graph to present to teams, showing them what to prioritize next.
This survey is modified from the original jobs-to-be-done model, which focuses on job steps and outcomes, but has helped me immensely when prioritizing insights for teams.
Overall, surveys help us along in our journey as user researchers, and these techniques have aided me throughout my career in giving teams the best information possible. Give these a try if you are struggling with how to prioritize during your next generative research study!
Nikki Anderson-Stanier is the founder of User Research Academy and a qualitative researcher with 9 years in the field. She loves solving human problems and petting all the dogs.
To get even more UXR nuggets, check out her user research membership, follow her on LinkedIn, or subscribe to her Substack.