Insights, Enhance! Using Product Analytics to Supplement Qual Research

The first time I saw a Google Analytics dashboard, I panicked. Although I had experience with quantitative data during my Master's program, for some reason, something about product and usage analytics felt utterly foreign. The person who was walking me through the system stopped and asked me if I had any questions. "How do you make sense of all this?" was the first of many questions that I pushed away.

Instead of getting familiar with quantitative data, I ignored it. I spoke about the importance of qualitative data and how it was much more informative and gave organizations more substance. But I knew I was down-talking quantitative data because I was scared of it, not because it was bad.

Eventually, I was fortunate enough to work with great product managers and product analysts who taught me the benefit of bringing quantitative data, specifically product analytics, into my practice.

The “sample size” problem, and a small concession

Qualitative research has the superpower of allowing us to dive deep into the mental models, unmet needs, motivations, goals, and pain points of our users. We get rich data filled with stories that can lead us to truly understanding what our users need to accomplish and building the right products to solve their problems.

However, there is always that tiny caveat to qualitative research—sample size. And, to be fair to all those colleagues we argue with, small sample size is an appropriate point. Qualitative data shines in telling us why people are acting in particular ways, or making certain decisions. However, qualitative user research is complicated to generalize to larger populations.

For example, let's say I work at a company that sells plants online to individuals and companies. After doing some qualitative research with about ten participants, I notice two main problems:

1. People are hesitant to purchase plants because they are worried they won't know how to care for the plants
2. Participants don't see how the plants will be shipped and are concerned they will arrive damaged due to past experiences with other similar companies

These are excellent problems to unearth and could be supremely helpful to the product team and organization. With a few quick changes, we could add care information for the plants and better explain our shipping process.

Armed with this information, I go to my team to report on these two nuggets, but I get the dreaded response of, "How do we know these are the most important problems?"

That is where quantitative data can help, specifically product analytics. This data can tell you the magnitude of a specific problem or behavior, reveal how many people are affected, and help you determine financial impact. By including key product analytics related to your qualitative study, you can help teams see the overarching effect of a problem across a more extensive user base and prioritize what is important.

One final perk of using product analytics is to tie your qualitative research back to the metrics and see how they change over time after you have changed the product. This process helps you measure the impact of user research on a product and organization.

Two tactics for incorporating analytics during qualitative studies

There are two main ways you can use product analytics in your qualitative studies:

1. Exploratory sequential design. This starts with qualitative research. Once the study is complete (or even when you start seeing trends part way through), you investigate your product analytics to see if there are variables to support or validate the qualitative insights on a larger scale.
2. Explanatory sequential design. In this scenario, you sit down (on your own or with an analyst) to determine which metrics are suffering. Are there key metrics (conversion rate, clickthrough rate) that are not performing well? You pull this quantitative data and then use qualitative data (either previous insights or running a new study) to explain further what is happening.

While all of this might sound simple, it certainly wasn't for me when I first started thinking about product metrics. If you are unfamiliar with data and trying to know what metrics to use with your qualitative data, you can ask yourself (or a colleague) the following questions:

• What type of actions would these insights cause people to do?
• What metrics might be indicators of these problems?
• Did people mention any behaviors during the interviews that might be important metrics (ex: keeping something in the cart for a long time)?
• What goals are people trying but unable to achieve, and how could we see this impact on metrics (ex: people want to purchase plants but don't have enough information)?
• If you solved the user problem, what impact might that have on the product? Which metrics would be impacted?

It has taken me quite a few years of practice (and lots of meetings with patient colleagues) to feel more comfortable exploring metrics, so don't get frustrated if it takes a few tries to get it right!

Example approaches and study designs

Let's stick with the example above of a plant company that sells plants online. In addition to the website we use to sell plants, we also have data coming in from an app that helps new plant parents track and take care of their plants.

Exploratory sequential approach

For the first example (exploratory sequential approach), we can use the qualitative data we collected from the study that highlighted two significant problems:

1. People are hesitant to purchase plants because they are worried they won't know how to care for the plants
2. Participants don't see how the plants will be shipped and are concerned they will arrive damaged due to past experiences with other similar companies

The next step in this exploratory sequential approach would be to look at some product analytics that would help us understand how widespread these problems may be. We dig into our Google Analytics platform, which tracks usage patterns and data across our website.

We consider what metrics might be indicators of the problems I described above. What would people be doing if they felt hesitant about care or shipping? Here are a few ideas on the metrics that the insights might impact:

1. The conversion rate might be low if people are hesitating to purchase products.
2. There may be a high amount of cart abandonment or an extended amount of time between add to cart and purchase
3. People may spend a lot of time on the "help" or product pages to find this information
4. Search terms about shipping or plant care would come up more frequently than others

When we dig into this data, we see that cart abandonment is high, and the most common help search terms include plant care and shipping, including a very long time on the help page in general. We can look at the percentage of users who abandon their cart and calculate the financial impact of these users not purchasing the plants due to these reasons. With this information, we have a better understanding of the potential impact of these insights and can move to make changes to help these metrics.

Explanatory sequential approach

Let's look at the other side of how to use product analytics with qualitative data. With this approach, we first look at product analytics to see what is (or isn't!) happening and then use qualitative data to explain why.

We have an app that helps people understand how their plants are doing by providing instructions and tracking simple functions (ex: watering schedules, sunlight hours, and plant health). We sit down to look at the usage data from this app and see that people aren't using the plant health tracking feature and that quite a few plants suddenly go from "healthy" to "dead" with no steps in between.

With this problem, we can potentially assume that those who suddenly log their plants as dead might not become loyal customers (which could show a drop in retention and customer lifetime value). These metrics are crucial to the company, but we still don't understand why this is happening.

After seeing these metrics, the next step would be to set up some qualitative interviews, ideally with participants experiencing these problems, to understand why this issue is surfacing. Through these interviews, we learn:

• Most participants don't know how to input plant health because they aren't sure of plant health indicators
• The plant health function has no clear system, while the others, they can easily track (ex: putting in date, time, and amount of water of tracking watering)

We gather the data we need to understand where to focus our resources to create new and more usable solutions instead of just assuming the problem through analytics.

Combining qualitative research with product analytics can give you a holistic picture of the user's journey regarding what is happening and why. This mixed-methods approach can lead you to properly prioritizing and focusing on the correct problems and solutions that help your users achieve their goals.