Issues with Data Gaps? Wrangle 'Em Together with Triangulation

Triangulation of data is a hugely important topic in user research. I first learned about it during my academic years but then shunned it for quite some time. I focused so much on primary, qualitative user research that I forgot all the other data that could be hugely helpful.

I rediscovered triangulation when I was struggling with a particular research study at a content marketing company. We were trying to understand why certain users didn't use most of our features—and why their general usage was so low.

Jump to...

• The challenge of filling the gaps
• Triangulation as the key
• How to bring data together
• Project-based triangulation
• Continuous triangulation
• A step-by-step approach

The challenge of filling the gaps

As you can imagine, trying to find and speak to users who aren't enthusiastic about your product or don't use it often can be challenging. That was the problem I came up against time and time again. I couldn't get hold of enough of these users to create any meaningful insights.

Not only that, but I needed to figure out where to focus my interviews when I could get a hold of them. It is challenging to find out why people aren't using your product often without outright asking them, potentially leading them to answer untruthfully.

I was at a loss. The team was disappointed. I had only managed to schedule three interviews. The first felt all over the place because I didn't know how to ask for the information my team needed.

After the subsequent two interviews and no more interviews scheduled, I knew I needed to get creative. So I returned to the drawing board and disregarded my standard toolbox of qualitative research methods.

Triangulation as the key

And that's when it hit me. I had to work with what I already had. So I brought together customer support tickets and data analytics. I interviewed the account managers to understand what these users talked to them about. All of this helped me paint a potential picture of what these users were going through.

From there, we made some positive changes based on this feedback, and I got more precise about what I wanted to focus on when we were able to schedule more participants.

From then on, I realized how important it was to look at other data sources in combination with primary research.

However, sorting and filtering through multiple data sources is messy, so I’ll share some strategies I use to bring various data sources together cohesively and in a way that complements your user research practice.

How to bring data together

I started this data collection project about eight months into my role as a user research team of one (and the first user researcher at the organization). So, you can tackle this at any time! Of course, setting it up takes time and effort, but it helped me so much in that role and became a framework I could bring to my following positions.

Before we dive into the step-by-step approach, the first thing to do is to set a goal (as always)! This goal will help you uncover precisely what you are trying to accomplish and narrow the initiative's scope. Not looking at all the data is crucial since that can quickly overwhelm you—at least, that was my initial experience.

When I started triangulating, I didn't have a goal outside of "figure out how to make this study work." For me, data is kind of like YouTube or Wikipedia. I went down many rabbit holes and spent hours completely sidetracked.

I could have spent entire days sifting through data, but that wasn't my only job. I had a research practice to run! So I decided to create some goals (and boundaries) around subsequent triangulation initiatives.

First, there were two ways that I triangulated data moving forward:

1. On a project basis, where I looked at other data sources to help me better understand the project I was working on
2. On a continuous basis, where I set up the framework for continually tracking different sources of data

Project-based triangulation

This type of triangulation is more common than the second (which I promise to dive into), but I want to cover it briefly.

I think of project-based triangulation as mixed methods research, and there are two main ways to triangulate project-based research:

1. Use a combination of primary and secondary data
2. Use two (or more) primary data sources

The most common way I have triangulated data is by using primary and secondary data together. I use this method because it’s faster than combining two primary sources.

For instance, if I'm running a usability test, I'll look at product analytics and an app like FullStory to see if the insights I've found from the test match up (or not!) with the other data sources.

This approach goes faster than conducting a usability test and following up with surveys and interviews. However, I love to combine two primary sources if I have the time.

Considering two primary sources, suppose I run a qualitative study like 1x1 interviews. In that case, I will complement that with quantitative data—either from a survey (two primary sources) or reviewing usage data (a primary and secondary source).

There are several ways to approach mixed methods data…

✔ Explanatory sequential design

An explanatory sequential design emphasizes quantitative analysis and data collection first—followed by qualitative data collection. We use the qualitative data to explain further and interpret results from the quantitative data.

✔ Exploratory sequential design

An exploratory sequential design starts with qualitative research and then uses insights gained to frame the design and analysis of the subsequent quantitative component.

✔ Convergent parallel design

A convergent parallel design occurs when you collect qualitative and quantitative data simultaneously and independently—and qualitative and quantitative data carry the same weight. They are analyzed separately and then compared and combined to confirm or cross-validate findings.

I choose between explanatory and exploratory sequential design based on what information I need upfront.

For instance, if we are trying to understand "why people go on holiday," I would start with a quantitative approach to narrow the scope of the qualitative element. It would be hard to approach such a broad topic with interviews.

On the other hand, if the problem space is more straightforward, such as, "How do people choose their destination for leisure travel," I would start with a qualitative approach and follow up with a survey to help validate findings.

I recommend making mixed methods a regular part of your research toolbox—it is a fantastic skill to have and can help your teams understand a more holistic picture of your users.

Continuous triangulation

Continuous triangulation is more tricky because you are looking at data on an ongoing basis.

When I started diving into all the data we had, I was initially overwhelmed (but excited). I had thousands of customer support tickets, hundreds of reviews, many data sessions, and a place where colleagues could put and vote on internal suggestions.

It was a lot, and after focusing on qualitative research for so long, I felt rusty and slow going through this data, even though some of it was qualitative-based.

So, I went ahead and created some goals and boundaries.

My boundaries

1. I was only going to look at data from the previous three months. Our product had changed quite a bit since then, and I would have spent weeks combing through all the data we had.
2. I would spend 2-3 hours per week looking at the data, and no more than that.
3. I would get others on board to help me (ex: data analysts and product managers) to sort through and categorize the data during bi-weekly or monthly meetings.

My goals

1. Focus on the top two or three most relevant and helpful data sources.
2. Create a taxonomy for incoming data moving forward that made it easy for the teams and me to sort through.
3. Match any new data with current or upcoming projects. The product manager would review the data to see if it impacted the requests for upcoming studies.
4. Get help creating a dashboard to pull important metrics forward to make it easier to review.

My step-by-step approach

Starting from scratch certainly wasn't fun, but it allowed me to experiment and learn new things. Although this approach won't be for everyone, hopefully it gives you a starting point to work from!

Here is my framework for continuously tracking data to use in your user research practice:

✔ Identify your data sources and skim through them

Understanding what is available to you and the data quality indicates what you can track. For example, if your product analytics tracking is messy, it's best to find other sources that will be more reliable.

✔ Pick the top two data sources that are generally most helpful

Find which data sources are the ones that give you the most beneficial information. For example, I occasionally looked at reviews, but they weren't as helpful as customer support tickets. Of course, this can change in the future, but it's best to focus on two to narrow your scope!

✔ Create a taxonomy (with others, if possible) to help you categorize the data

I start using global tags, such as pain points, goals, and needs, and then I dig deeper into adding more specific tags to the initial global tag. For instance, if a participant had trouble with travel insurance, I might use the tags: #painpoint #travelinsurance

✔ Choose a date to start from

You don't have to go through every piece of data from years ago. Instead, choose to start from three or six months prior. I usually start looking at data from about three months before.

✔ Set the number of hours

Decide how many hours you plan to spend each week sifting through the previous and new data. I recommend starting with two or three hours a week (if you can), but do whatever feels right to you.

✔ Bring others in to help you

Go through the data and set a bi-weekly or monthly meeting where you sift through the data together. This step is crucial because you don't want to be the only one holding this knowledge!

✔ Go through the data and categorize

I use a Miro board and affinity diagramming (with a few layers) for this. I set my standard global tags and reviewed the data, adding stickies to the global tags. For instance, I would put all the relevant customer support tickets into pain points. I then break down each global category into more specific tags. For example, I would take pain points and create smaller categories for specific pain points, such as travel insurance, delayed trips, last minute changes.

✔ Start applying the data to current or upcoming projects

I use either a Miro board or Google Sheets for this, but you can apply this to what your teams are using. I look at the current and upcoming research projects and link back to the relevant data I found in the step above. If my colleagues aren't working with me on this, I will send them the links to data to review and see if that changes any upcoming requests or research questions they have.

✔ Get help with creating a dashboard

If you are using analytics that a colleague can quickly put into a dashboard (ex: conversion rate, click-through rate, app downloads, satisfaction metrics, etc.), kindly ask someone to help you create a dashboard.

I use this dashboard and the above Miro board to review incoming data and apply it to upcoming projects. Additionally, this allows me to see if anything is going wrong that we’re missing that we could turn into a research project.

✔ Rinse and repeat the process

As I said, I spend about two or three hours on this data per week and have bi-weekly meetings with my teams to sort through it!

Overall, both project and continuous triangulation are essential. Although project-based triangulation is much easier and a great starting point, continuously tracking and using data for your research projects is a huge superpower. It takes time to set up and find your organization's exact framework, but it can help you and your teams immensely in the long run!