Relevance Trumps Usability

Last week, a customer told me that he learned more from a user in 30 seconds than he does in a typical 9-minute usability test.

How? He asked users this one question — one that people don’t ask in usability tests.

In the typical usability test, we spend hours recruiting and screening users, defining business goals, creating tasks, composing scripts, and crafting prototypes, all while dancing around what we really care about: Does this user find my product to be relevant in their life?

We try to get at this question by doing a usability test. But usability tests are often abused; whether someone can achieve a task within your product is not nearly as important as whether they see themselves using it.

Can they use it? That’s the question usability tests try to answer. But will they use it? That’s something else entirely.

The average usability test goes like this:

You tell a grocery shopper to buy ingredients to make cookies, and to pick the chocolate, milk, eggs, flour, in a particular order. You’ve given this shopper a discrete set of instructions, because you want them to mimic the behavior of a super grocery shopper. This test shopper is not allowed to roam freely and expose your product’s innate issues. Is this user penalized if they find flour before they find chocolate?

In the average usability test, you ask users to comment as they click through your product. You give specific instructions so you can see where users stumble as they go down the task list. The user is too busy to see whether the product has relevance in their life.

Results nearly always come back skewed, because the user wants to follow your instructions and perform well for you. This is a

Now imagine if you let go and simply observed users. What if you invite another grocery shopper to buy the ingredients to make cookies, and then stop giving instructions? With free reign to roam, the user might poke around your grocery store, find some ingredients, and perhaps miss others.

You’ll gain empathy with users by observing their behaviors. You can find patterns, and unlock opportunities to improve your user experience. But even by observing, there’s still one missing piece: a reflection on the experience.

When push comes to shove, a new user needs a few minutes to explore your product or prototype, and only needs 30 seconds to tell you whether your product is relevant to them — and why.

Snapchat is a perfect example.

Wired claims that Snapchat’s Missing Out on Millions Because It Sucks to Find Stuff on Its App. Even four years after launch, you’ll find dedicated Medium posts explaining how to use Snapchat. The home screen is a camera. A new user to Snapchat has to guess what to do next: swipe left and right to unlock the nuances between four content pages, swipe down to access their profile, and not bother to swipe up because nothing’s there.

So if Snapchat’s so hard to use, why does it have over 100 million daily users? Because it’s relevant to people’s lives. Snapchat makes it easy to share even the dullest micro-moment without consequence. Teens love it. They master it and know how to use filters, decipher emojis, and add text to content. Parents? They just want to send photos.

A usability test can compare how teens and baby boomers navigate Snapchat, but it won’t address this looming question — is Snapchat relevant in my life?

Even if it’s hard to use, Snapchat’s relevant to 100+ million people every day.

So how do you figure out whether a product is relevant in just 30 seconds? Ask this one question: In thirty seconds, share one hit, one miss, and one wish you have for this product.

You’ll learn in 30 seconds what a usability test might show you in 9 minutes: whether your product is relevant to the user. 30 seconds forces the user to only discuss features that matter to them — the features that your team should be laser-focused on improving, removing, or building. Less is more.

What do you learn from hits, misses, and wishes?

• Hits: Discover marketable features that resonate with users. Build around these for growth.
• Misses: Expose where your product is weakest in the user’s mind. Improve these to reduce churn.
• Wishes: Inform your roadmap by unlocking the features people think should come next.

These thirty seconds will also reveal what people don’t talk about. Most usability tests focus on watching users try to find advanced features, such as filters on Snapchat. Why wait 9 minutes to see if someone can find an advanced feature when you’ll know in 30 seconds if it’s relevant? If advanced features are really central to your product experience, yet nobody discusses them, make them easier to access; if they’re not core to the user experience, ditch them.

So when your team says they want to run another usability test, first ask yourself: are we after usability or relevancy? Then figure out what kind of test you need.