Mixed methods research practices

1. First click testing 2. Tree testing 3. Card sorting 4. Quantitative usability testing 5. Product analysis

Ideas

Elevate Your Mixed Methods Research with These 5 Practices

It’s easy to get attached to exclusively qualitative methods. These quant ideas will help bring your practice to a more sophisticated level.

Words by Nikki Anderson-Stanier, Visuals by Allison Corr

I used to be a qualitative user researcher through and through (I call us “qualies”). In the first few years of my career, I wouldn't touch quantitative research with a 10-foot pole.

I had had experience with it through my master’s program, but something about mixed methods felt tedious and unexciting.

All I wanted to do was flex my qualitative muscles and talk to every user on the planet. I was (and still am) fascinated by how people thought and how they explained those perceptions.

In a way, I was scared of quantitative user research because I didn't understand it, and numbers never made much sense to me. The last thing I wanted to do was try quantitative research and look like I had no idea what I was doing.

After getting laid off, I faced my fear of looking like an impostor and went to work up-leveling myself to be more equipped for mixed methods user research.

New approaches

Once I learned surveys weren't the only quantitative approach people cared about, I searched (nervously) for others. But, as a quick caveat, surveys are hugely important, so feel comfortable writing them!

There were a few methods that caught my eye:

First click testing
Tree testing
Card sorting
Quantitative usability testing (benchmarking)
Product analytics

Although I was terrified of including quantitative user research and numbers in my process, I knew I needed to become a better researcher, and this was a path forward. I went ahead and started to read everything I could about these methods and practiced them in any way I could.

1. First click testing

First click testing helps us look at what a participant is clicking on first on an interface to complete a given task. It allows us to examine how effective our product is in helping users achieve tasks.

If the first thing people click on to complete a task is incorrect, they could potentially abandon it. You can test first clicks with a live product, prototype, or wireframe.

For example, imagine we were working at a furniture company, and we wanted to assess how people found the type of furniture they wanted. We could use first click testing to see how they would begin searching for a given piece of furniture (ex: a couch). This test would indicate if something might confuse or lead participants down an incorrect path.

These are the steps I take to set up and run a first click test:

Create tasks focusing on a problem they must solve rather than telling them where to click. For example, "You are trying to find a yellow armchair for the corner of your living room..."
Ensure there is a happy path(s) for each task and that you have aligned with your team on the happy path(s). This will make it much easier when it comes to analyzing the data.
Track where participants click when you give them the task.
Time how long it takes the user to make this click. Taking a long time to make that first click may indicate a navigation issue or confusion.
After each task, you can use the single ease questionnaire (SEQ) to track the perceived usability of the task.
Optionally, you can also ask about confidence on a 7-point scale to assess their perceived confidence in completing the task correctly.

I run first click tests unmoderated to get a larger sample size of data. Using an unmoderated tool also makes tracking clicks and time on task easier, since many tools have those metrics baked in.

2. Tree testing

Tree testing was the second quantitative method I picked up, and I was super happy when I learned about it. Before that, I had always wanted to run a card sorting test, but it often felt too expensive and challenging. Tree testing was a great answer to that.

In a tree test, participants attempt to complete tasks using only the category structure of your site. This type of test enables us to understand if people can use our information architecture to find information to complete their tasks.

If we go back to the example of working in a furniture store, the category structure of our website might look like this:

Furniture

Bedroom
Kitchen & Dining
Office
Outdoor
Living room
- Coffee tables
- Cabinets
- TV sets
- Seating
  - Sofas
  - Bean bags
  - Recliners
  - Armchairs

We could ask participants a similar question to the above to find an armchair, and they would have to use our current structure to navigate to the appropriate page. Analysis of the tree-test results will show whether people could find the right path to this item in the information hierarchy.

The steps for setting up and running a tree test are very similar to the ones I outlined above for the first click test, so if you want to run this type of test, you can refer to that!

3. Card sorting

Card sorting is a grey area because some people think of it as qualitative and others as a quantitative approach. From my side, it depends!

Open card sorting is great for exploratory work and understanding how users relate to, organize, and define different concepts. In addition, it can lead to a better understanding of terms and definitions. In this approach, participants create categories and concepts of their own and then order them. Open card sorting is qualitative.

However, closed card sorting is a more quantitative approach. Closed-card sorting is when participants get cards with information already written on them and are limited to using them. This approach is very evaluative and is best when the terminology or concepts are well-defined and established. Then, it can give you apparent patterns on the cards.

To read an in-depth guide on setting up and conducting a card sorting study, check out this article.

4. Quantitative usability testing (benchmarking)

Quantitative usability testing can show a product's performance on a given set of metrics.

Typically, those metrics look at the following:

Efficiency: Whether or not people can complete tasks on your product.
Effectiveness: How long it takes someone to complete tasks on your product.
Satisfaction: Their perceived feeling about how your product meets (or doesn't) their expectations.

Quantitative usability testing is similar to qualitative usability testing in asking participants to complete tasks. However, while qualitative usability testing is based on observation and identifying issues, quantitative usability testing focuses on collecting metrics like time on task, task success, and perceived usability.

You can use quantitative usability testing to measure these metrics once, or you can use them over time to track the progress and improvements in a product. If you use them over time, this type of study is called benchmarking.

To read an in-depth guide on setting up and conducting a benchmarking study, check out this article.

5. Product analytics

Diving into the world of product analytics was the most scary for me. I didn't understand what metrics to look at or what they meant. But, over time, I learned what these metrics are (they are dependent on your organization and business model) and how to use these dashboards.

Product analytics can help you identify problems that you can further research or monitor how something on your product is performing.

For example, if we look back to the example of working at a furniture company, we could go into our web analytics to search for essential metrics or flows. One of those might be understanding the flow of someone filtering through the search results. We could identify what people are doing with our filters and where they fall off.

Similarly, you could also look into a checkout experience and where people are dropping off, indicating an issue with the flow.

Once we see what is happening and identify potential issues, we can conduct some qualitative (or quantitative!) usability testing to figure out what is going wrong.

On top of that, after we release a particular feature, we can track its performance through these metrics to understand how people are using it or if there might be issues with it.

Collaboration is key

Although we can learn quantitative research and approaches, another thing to do is look around for people who can help you. Just like we’re experts in our field, they are in theirs, and they might be able to support you better than you can imagine.

For example, when I got into product analytics, I was perplexed and overwhelmed by the Google Analytics dashboard. So instead of getting stuck, I asked a product analyst to walk me through how to use the dashboard several times before I got the hang of it.

There are so many people in organizations with skills beyond our knowledge, and it is so important to tap into those colleagues to up-level ourselves (and teach them something in return)!

Learning mixed methods research is crucial moving forward. Although we can still specialize and be more comfortable on one side, it's hugely advantageous to be flexible in our approach to problems. Quantitative user research gives us a whole other side to explore and opens our minds to new ways to identify and solve team or organizational issues.