The "Highs & Lows" Approach to Product Testing

User experience insights professionals are always on the lookout for a reusable tool.

Be it an email template, question bank, an interview guide, or a deck skeleton—the tricks (or treats) of the trade help us inject rigor into the discipline and keep us two (OK, maybe one) step ahead of our deadlines.

In the spirit of Halloween, we decided to conduct some original research on candy (who doesn't love a good ranking study?). The design we use here to illustrate the “best and worst candies” can easily be emulated into a powerful framework for product testing—one that serves up empathic feedback on your product’s "highs and lows."

Let’s take a look at the study design. We’ll learn the extent of the ire people have for candy corn. And we’ll pick up a framework for identifying “would-be” product issues, and surfacing unnoticed moments of product delight.

Design vitals

In a “Highs and Lows” mission, researchers ask participants to list the standout highlights and pain points of a product, an experience, or anything else they want to take stock of (the "high" and "low" moments, as it were).

At its heart, it’s similar to an inventory mission. But instead of inventorying someone’s home, toolkit, kitchen, etc., you’re asking them to inventory their own preferences or frustrations.

In a single line of questioning (Diary entry, Express mission, or set of questions), ask your participants to name something that sticks out to them as good or bad about your product, and probe a small amount on the item they list. At the end, you’ll be left with piles of the good, the bad, and the ugly in your product.

The primary benefit of a “Highs and Lows” study is that it’s a truly bottom-up approach to understanding what’s working and not working in your product. Instead of trying to make hypotheses about pain points and then testing them with your participants, you’re directly asking them what stands out in their everyday experience.

Highs and Lows are a good fit when you don’t quite know what you’re looking for, or for discovering unknowns about your users.

It’s also versatile. The approach can be a great open-ended deep dive into your product, or a light, quick-hit study. A light touch High and Low can give you a birds-eye view of your users’ experiences and help determine subjects for future deep dives and testing.

We love Highs and Lows as a light lift design because it’s a big payoff for very little effort, both for you and your participants.

For your participants, they’re given an easy question. It doesn’t take a lot of work for people to think of something they love or hate about something they use. And because you’re asking specifically for stand-out experiences, people often have a lot to say and are ready to share their opinions.

For researchers, this is also a very straightforward study to both program and analyze. It can be as simple as counting mentions of specific product features to get a sense of stand-out joys and pain points, but can also leave room for a lot of nuance if you have the time.

For our example, we looked to find out the "highs and lows" of the candy world. We asked people to sound off on their favorite and least favorite candies and tell us why they like or dislike them.

The data came back in about 24 hours, and with a light touch of analysis, we were able to get a good sense of the landscape of great and terrible candy.

Design template

The "Highs and Lows" design is a flexible framework that can be tweaked to suit your needs.

The first decision you need to make is who you want to recruit to ask about your product. Are you looking for in-depth feedback about all the nooks and crannies of your product? Use a screener to recruit some power users.

Or, perhaps you’re curious about what stands out the most to casual or first-time users? Then consider finding people who have never used your product before and asking them to do first impression highs and lows—what immediately stands out after just a few days (or even a few minutes) of use.

You can even ask about the highs and lows of competitors instead of your own product. In this case, it doesn’t matter whether you screen for product usage or not, and ask people to share whatever product is relevant to their own life.

Next, you’ll need to decide what your “Highs and Lows” prompt will actually be. If you’re interested in the general state of your product, you can go as broad as you like and leave people wide open to tell you anything and everything:

• “Tell us one highlight about using this product.”
• “Tell us one thing that bothers you about using this product.”
• “Tell us your favorite thing about using this product - it could be a feature, an aesthetic, an experience, or anything else.”

Or, if you have a more pointed question that you want to explore, you can give some more specific guidance on the kind of high and low you want to inventory:

• “What’s one thing that’s really working for you about checking out of the website?
• “What’s one thing that’s not working for you?”

You can also play around with framing of “highs” and “lows.” Try:

• Roses and thorns
• Joys and frustrations
• Highlight and lowlight
• Favorite and least favorite

You can ask people to tell you just highs, just lows, to choose one, or to do one or more of each. In our study, we asked people to choose one, and went with “love” and “hate.”

Once you’ve decided your basic framework, it’s up to you how deep you want to dive. If you’re only interested in finding out what highs and lows are top-of-mind for your participants, you don’t need a lot of detail.

For example, our candy study was run entirely on Express (our media-rich survey tool) for a light touch and a fast turnaround. But if you wanted to do this more in-depth, you could put this into a Diary study—or any longitudinal study that allows people to report their highs and lows over time, or when experiencing a particular “trigger.”Regardless of how deep you want to dive, here is a starting point for the kinds of questions you might want to include in your Highs and Lows design:

Headline question

The first question should just be a headline of what you’re working with. This will be a way for you to easily count and sort the themes that arise from the study.

Do not ask them to elaborate here as it will make the data more difficult to work with. Ask them to simply give their problem a name, and then dig deeper later on.

Exploration question

Ask an open end or video question to ask why they listed this topic as a high or a low. Keep it short and light. It will give you some sense of the reasoning, but won’t be overwhelming to analyze. Use it to inform future deep-dive designs.

Since you’re trying to get people’s strong reactions to this topic, feel free to get creative with your video prompt to elicit more emotion!

Close-end metric question(s)

Multiple choice questions are your friends in a light-touch study like this, since they require minimal effort for both you and your scouts. We asked about the qualities that stand out in good and bad candies, but other questions can easily be slotted in here, such as:

• What part of the product does this Low affect?
• Out of the following list, which best describes your emotion when encountering this issue?
• What words would you use to describe this Low?

Improvement question

This, like the video question, will give you an initial idea of what’s bothering people and how they envision your product improving, which you can follow up on in future studies.

Extra questions

In the context of a longitudinal, or more robust study, you may want to dig deeper into each of these highs and lows for more context and nuance. Consider adding questions like:

• On a scale of 1-5, how much does this high or low impact your experience?
• How often do you run into this high or low?
• What contexts do you run into this high or low?
• How does this high or low make you feel about the product?
• How would your experience be different if this high or low weren’t there?
• (If moments-based): where are you and what are you trying to do? What caused this moment to occur?

Analysis plan

Analyzing a Highs and Lows mission, like any qualitative-focused mission, can be daunting at first. This can especially be the case if you decided to use a longer study to collect multiple highs and lows per user, which generates a mountain of data to work through.

Here are some places you can start to begin making sense of your data with minimal time spent.

1. Tag and quantify headlines
2. Describe the highest highs and the lowest lows
3. Find cross-high and cross-low themes
4. Build deliverable assets

Tag and quantify headlines

Tagging your “headline” data should be your first stop on any Highs and Lows mission. Your job here is to examine the short “headline” question you asked, develop a tag list based on what people mentioned, and apply tags to that response.

For us, this simply meant turning people’s favorite and least favorite candies into a tag list:

Once you’ve tagged all your answers, you can begin by comparing frequencies of different tags to see if there are any stand-out ‘winners’ or ‘losers’ in your product. This in and of itself can be an important finding.

For example, we found that Reese’s are a clear stand-out favorite.

While candy corn is strongly disliked.

Describe the highest highs and the lowest lows

Once you have your data tagged, you can use those tags as an organizational tool to drill into specific Highs and Lows that are of interest.

Filter to the relevant tag and explore the entries, concentrating specifically on the open-end follow-ups you asked about why those things are so good or bad.

I suggest starting with the top three highs and the top three lows that you collected, since these are the things that your participants have agreed are most salient in your product. You can also ask stakeholders if there’s something on the list that’s of particular interest to them, and start there.

In this study, I filtered to those who love Reese’s, which was the most-selected favorite candy. I found that scouts love this candy because, “Peanut butter is delicious.”

Explore wider themes

Some of the most interesting insights from High and Low studies can come not from the “what’s” (i.e. what features ranked highest and lowest) but from the “why”s (i.e. emotions that are associated with highs and lows, common elements of highs and lows that motivate them being exciting or frustrating).

To explore these, try looking across several of the highest and lowest rated entries you collected and see if any themes stick out across the board. You can use your exploratory open-ends to find patterns, or make use of your close-ended metrics.

You can also try comparing answers between all highs and all lows. What elements are present in high moments that aren’t in low moments, and vice versa?

For example, in looking at all good candy data vs. all bad candy data, I found that texture is a key driver of good and bad candy experiences, regardless of what particular candy people selected as the best or worst.

Crunch/crispy (53%) and creamy (37%) are the most desirable textures...

54% of worst candies are chewy

Build deliverable assets

Now that you’ve found some interesting insights—whether they're about particular features or about the cross-feature motivations—you can use your qualitative data to build powerful assets to communicate those findings to your stakeholders.

Video data is a particularly impactful way to get your points across. Filter to the feature you want to communicate about and bookmark some particularly impactful videos that represent your wider findings. You can share individual videos or even showcase entire entries. You can also create reels from multiple videos to communicate how wide-spread a theme is.

For example, I might profile a star scout explaining why Kit Kats are so good. I select a scout who echoes many themes I’ve heard throughout the study: texture, chocolatey-ness, and variety. I’ll introduce her and show a video:

“Anna A. Loves Kit Kats because they’re shareable and fun, and they always keep their brand fresh with new flavors.”

For one of the most hated candies, I thought it would be more fun to profile multiple people, so I put together a quick 1-minute video in dscout’s video builder. When I share these, viewers get a quick-hit sense of why scouts find these candies so unpleasant to eat.

“Highs and Lows” is an excellent tool for product testing, or just getting some quick insights on what your users think about a particular product or experience.

If you’re looking for more strategies for digital product testing, take a look at our People Nerds Guide.