Turning to Anecdotes as Allies

“Anecdotes are my worst nightmare.”

It’s a phrase we hear time and time again in the UX research community. We fear that a Product Manager or other executive will join one customer call, hear one complaint or excited request, and rescope a roadmap overnight.

But…what about my synthesis? What about the trends?

We have good reason to approach anecdotes with a wary eye—the example above isn’t an exaggeration. It happens frequently, especially in democratized settings. But the antagonism that’s taken root misses how anecdotes can empower research teams and add impact to the learnings they generate.

Let’s turn anecdotes into our allies.

Taking a deeper look at anecdotes

To form our alliance, we must first understand the origins of our new friends:

Anecdote:

1. a short amusing or interesting story about a real incident or person
2. an account regarded as unreliable or hearsay

From the Ancient Greek ἀνέκδοτος, meaning “accounts unpublished”

Perhaps we’ve been conditioned against trusting anecdotes since childhood. As we grow up into professional researchers, we regard these “amusing, unpublished accounts” of “unreliable hearsay” more and more as forces working against our pragmatic, empirical approach.

What data-driven product and business leaders want to make decisions based on unreliable hearsay?

Yet, after a recent experience, I’ve come to view this ancient definition as particularly empowering.

My team was on a roll—we’d recently completed a string of rigorous, deeply insightful studies that should have had a positive impact on a number of key company decisions. Yet the learnings weren’t reaching the folks who needed them.

Research teams often face this challenge, and there are many areas our team needs to improve: more inclusion, more brevity, more collaboration, more political savvy. But as a team of two, we had our hands full.

I sought out a few senior leaders to try to understand the root of our challenge, and quickly identified a key gap in our understanding: the team wasn’t placing value on formal insight synthesis (i.e. capital-R Research deliverables), and rather was valuing anecdotes. Our leadership believed that our engineering, design, and product teams needed to talk to users more, and that what they learned would lead to delightful solutions. And specifically, they believed that because actioning on anecdotes is human nature, trying to train a different muscle would be futile.

At the high-level, this is a relatively innocuous position. We want our colleagues to be more user-centric and to spend more time listening and learning from users, and it's true that humanity’s system of learning and growth is built on anecdotes. But when one-call-per-week—or NPS survey comments piped into Slack daily—is the preferred action-driver over rigorous analysis of insight trends, neither the company nor our users will be best served.

Why is this exactly? Part of the answer is in the definition we’ve discussed: unreliable. One thing that one person says might be reflective of a broader sentiment. But how would we know?

A user might tell you that feature B is useless, and that they want you to build features C and D instead. She might tell you that she’d be willing to pay a lot for features C and D (“I’d buy it tomorrow.”).

Well… maybe. I know my leadership would agree that acting on this type of anecdote isn’t the right approach. They’re probably reading this thinking, “He doesn’t get where I’m coming from at all!”

The rest of the answer, however, is the key.

Anecdotes are seductive lures to the passionate mind. When we want something to be true, we seek out evidence that agrees with our perspective. This often happens subconsciously. Frankly, it often happens consciously as well. It’s selective listening, leading questions, and confirmation bias. It’s saying, “I just talked to a customer, and they said my plan is exactly what they want.”

Practicing selective listening and anchoring on anecdotes can lead companies on expensive goose chases that only become apparent in retrospect. And it’s not just that they can—they do. All the time.

Tips for better utilizing anecdotes

How do we dodge the goose?

First, we recognize anecdotes for what they are: a great way of building empathy and understanding of user experiences, sentiments, and mindsets. The actions they should drive are more engagement and listening.

Second, we recognize that when we combine anecdotes, we find trends. Trends are actionable, and are the unit we use to drive healthy decision-making. More engagement and listening nets more anecdotes, which become trends, which ultimately serve as inputs into strategic action.

These aren’t necessarily easy concepts to teach, and it can take time to change cultures that aren’t yet familiar with them. Here are a few tactics to aid the process:

Identify anecdotes in real time: When you notice someone anchoring on an anecdote in a conversation, chat, or doc, try commenting, “That’s an interesting anecdote! Do you think it’s a trend across other users?” By asking the question instead of posing a challenge, you prompt a dialogue.

If your partner says yes, you can ask why, which leads either to them admitting they’re working off a personal assumption, or to a stronger mutual curiosity to discover if the trend holds up. If they say no, you’re in a similar place; if the company chooses to proceed on assumptions, at least it’s with eyes wide open.

Catalogue and share instances of anecdote anchoring: You’ll probably have a few of these conversations in short order. Write them all down, and share them with your stakeholders. In other words, synthesize the trend! Grouping instances together can help people recognize them in real time and be more aware when speaking to users or analyzing user data.

This is especially effective if you catalogue not only the instance, but also the consequences. What decisions stemmed from the anecdote? This is a great contribution for researchers to make at retros (hopefully you hold them often!).

This works whether you noticed a trend that ran counter to the anecdote (“Our analysis suggests that this wasn’t the accurate insight to take away.”) or that supported it (“This turned out to be a trend, and here’s how we determined that.”).

Present anecdotes with other data types: Pairing or contrasting anecdotes with product usage or marketing metrics can both help humanize those metrics, and highlight misleading anecdotes. An anecdote pulled from a trend can help your team understand what a quantitative metric “feels” like to a user, while a one-off can be readily exposed as an outlier when shown next to quantitative trends.

Find allies and spread the gospel: For research teams to succeed, they can’t become lonely voices. It’s certain that there are folks on your team that will share your perspective—arm them with language and data to spread best practices when you aren’t in the room. You won’t avoid every goose chase today, but maybe in a year, instead of a whole flock the company will just be chasing one straggler.

The engagement generating power of anecdotes

One thing my VP often says is to focus on solving problems, and I set my mind to solving the challenge our team was facing. My approach was to involve far more colleagues in my next study, and encourage them to regularly post anecdotes from the study in a public Slack channel.

If we could tangibly show how the anecdotes they heard from a specific user did or did not match up to the trends identified across the full study, we’d be on a healthier path moving forward.

Over the month that we ran the study, over 100 new teammates joined the Slack channel (which had been around a while), discussing emerging patterns and pushing back on each other’s anecdote-anchoring. Nearly fifty team members shadowed user interview sessions (up from a maximum of five in any past study). And our study review presentation—where we discussed insights and flagged out a few particularly seductive anecdotes—had the highest attendance yet.

Counterintuitively, the proliferation of anecdotes we supplied led to more appetite for and engagement with rigorous empirical insight synthesis. We turned our Slack channel and user interviews into a trojan horse for our primary objective.

Will some of those fifty observers anchor on an anecdote they heard? Probably. Are our leaders demanding insight synthesis to back decisions? More than yesterday.

UX Research’s journey to influence and impact is often a long road, so let’s take all the help we can get. I, for one, am thankful for my newfound ally.

Long live anecdotes!