When Usability Testing Has Done Its Job, Consider Visual Testing Next

I used to say no to any visual testing requests. To me, usability reigned far over visual testing. If our product wasn't usable, our users wouldn't care how aesthetically pleasing our product was.

Many products don't have the essential components they need: functionality, reliability, and usability. So, whenever my teammates asked me about conducting visual tests, I turned each one into a usability test. But I eventually hit a wall.

We had done many usability tests and were in a good place, so visuals were a natural next step. There was no reason to say no, so I tried to test visuals in a way that wasn't biased, leading, or pointless with a smaller sample size. But first, I had to learn the actual goal of visual testing.

Jump to

• What is visual testing?
• When does visual testing make sense?
• But first, usability testing
• Visual testing methods
• Goal: Discover initial reactions and impressions of a design or brand
  • Method: Five second tests
  • Method: Aesthetic explanation

• Goal: Understand how people perceive brand or design attributes
  • Method: Open word choice
  • Method: Closed word choice
  • Method: Quality rankings
• Goal: Learn how visuals impact people's experiences and behaviors
  • Method: First-click tests
  • Method: Tree testing

What is visual testing?

Whenever I think about visual testing, I start with what it isn't because that is the most crucial part.

With visual testing, we don't focus on:

• Figuring out if one design is "better" than the other
• Asking people to pick which design they prefer
• Asking people if they like the design or not
• Discovering if people like the colors
• Understanding the impact of minimal visual changes (this is A/B testing!)

It’s easy to fall into the trap of asking these questions and relying on user research to answer them. However, user research can't answer these types of questions.

There are a few main reasons why qualitative user research doesn't help answer these questions:

1. We use a small sample size, and asking seven people opinion-based yes or no questions is not representative of a population. It won't lead you in a valid or reliable direction.
2. People have no idea what they want. For example, have you ever been asked whether you prefer different colors in an app? Or which design you like? It's a challenging question for people to answer. So instead, we focus on their needs.
3. Opinions are a lot less helpful than perceptions or reactions. Asking a small group of people if they like something won't help you understand their needs, goals, or pain points and could result in unactionable data.

When does visual testing make sense?

Here are the goals I try to achieve when using visual tests:

• Discovering initial reactions and impressions of a design or brand
• Learning how visuals impact people's experiences and behaviors
• Understanding how people perceive brand attributes

As you can see in these goals, we aren't trying to have people necessarily choose a preference or a design, but instead looking at their perceptions. That is a critical point.

I want to stress how important this point is. We have to think about what these goals and insights mean when we are setting up and reporting on visual tests.

Many stakeholders want to know "if people like something" or "which design to move forward with." Unfortunately, we can't twist the results to answer these questions directly. Therefore, when we go into visual testing, we must align with stakeholders on the fact that we will gather perceptions and reactions.

We won't be able to say, "this design is it" or "people loved design A," at the end of these tests. Instead, we will be able to explain how people reacted to and how they might describe each design, the good and the bad. Then, we can make a more informed decision moving forward.

But first, usability testing

Before I jump into the different visual testing methods I've successfully used, I want to bring up usability testing.

Aaron Walter created a hierarchy of user needs that mirrors Maslow's hierarchy of human needs. In this diagram, he stresses the hierarchy of how a product relates to users' needs.

The reason I bring this up is that the first three needs are:

• Functional
• Reliable
• Usable

Only when we fulfill these three can we build a more delightful experience.

So, when we think about visual testing—which generally looks more at the perception of delight and joy—it’s essential to assess whether we are achieving the first three.

That's why I tend to default first to usability if I ever get a visual test research request. Because as visually appealing as a product is, no one will like it if it isn't functional, reliable, and usable. So focus first on the usability aspect and then address the visuals.

Want a step-by-step guide to usability? Check out this article!

Visual testing methods

Okay, so we've done some usability testing and are ready to get into some visual testing. Where do we start? With the goals, of course! Here are the various methods I use, depending on the goal you are looking to achieve.

Goal: Discover initial reactions and impressions of a design or brand

This goal is about seeing people's initial responses to a brand or visuals. The main methods I use for this goal are:

Method: Five second tests

This method allows you to measure how well a design quickly communicates a message.

Use it to learn:

• Do you users get your main message or not?
• Does your brand look trustworthy or not?
• Do people understand your main product or not?

How to set one up:

1. Define a problem or need that needs to be solved
2. Brainstorm possible designs and copies
3. Create 5 to 10 different variations
4. Write a script of questions
5. Show each user different variations and ask them the questions

Script examples:

• What do you think that website was about?
• What product or service do you think that website offered?
• What was the most prominent element on the website?
• How would you start a trial/demo?

Method: Aesthetic explanation

Aesthetic explanation lets you understand how someone would describe their initial impressions of a design or brand.

Use it to learn:

• How do people describe your brand or design?
• What do people focus on when it comes to your brand or design?

How to set one up:

1. Define what you want to understand about a design or brand
2. Create 5 to 10 different variations
3. Write a script of questions
4. Show each user different variations and ask them the questions

Script examples:

• How would you describe this brand or design?
• What stood out to you about this brand or design?
• How would you describe this brand or design to a friend/family member/colleague?
• What was confusing about this brand or design?

Goal: Understand how people perceive brand or design attributes

Within this goal, we are trying to understand how people perceive a brand or a design. One of the best ways to do this is to have them describe their perceptions. I use the following methods:

Method: Open word choice

Allows you to understand how users view and describe your brand.

Use it to learn:

• How do users perceive and describe your brand or design?

How to set one up:

1. Choose (or create) several designs that you want to test users' perceptions of
2. Present the designs to participants
3. For each design, ask the participant to list several words that describe the design
4. If the participant only lists positives, probe for negative or neutral descriptors as well

Method: Closed word choice

Allows you to understand how users view and consider your brand attributes in relation to your design.

Use it to learn:

• Do users agree with your brand attributes?
• What brand attributes resonate more with users?

How to set one up:

1. Brainstorm brand attributes (both negative and positive)
2. Choose (or create) several designs that you believe encompass these attributes
3. Give users the list of brand attributes and ask them to select
4. Allow them to see the design as they choose words that resonate with them

Attribute examples:

• Empowering
• Approachable
• Disconnected
• Friendly
• Irrelevant
• Patronizing

Method: Quality rankings

This method helps you understand how users rate your design and brand against specific attributes.

Use it to learn:

• Do users agree with your brand attributes?
• What brand attributes resonate more with users?

How to set one up:

1. Pick 3-5 attributes that most resonated from previous research
2. Choose (or create) several designs that you believe encompass these attributes
3. Give users the list of brand attributes and ask them to rate (from 1-5)
4. Allow them to see the design as they rank the attributes

Goal: Learn how visuals impact people's experiences and behaviors

For me, this goal is the most important. It’s great if people respond well to your brand and design and rate positive attributes high.

However, I have seen many instances where visuals impact the usability and experience of a product. So, this goal is fundamental to include in your visual testing. Here are some approaches:

Method: First-click tests

First-click tests help you measure if users can do a specific task efficiently and effectively on your site.

Use it to learn:

• Can users complete necessary tasks with your design?
• Can users find where they need to go quickly?

How to set one up:

1. Define a problem or need that needs to be solved
2. Brainstorm necessary goals and tasks
3. Create three to five different designs for the tasks
4. Write a task script that includes the necessary tasks and make sure you give context and a relevant problem to solve (use different language than in the UI)
5. Run with 25-30 participants (unmoderated or moderated)

Task script example:

• Get into your account
• Find out how much the product would cost
• Locate what features the platform includes

Method: Tree testing

Tree testing helps you measure if users can find specific information on your website where they expect it to be.

Use it to learn:

• Can users easily find information with your design?
• Can users find what they need efficiently and effectively?

How to set one up:

1. Define crucial information that users need to find on your website
2. Brainstorm necessary goals and tasks
3. Create three different designs for the tasks
4. Write a task script that includes the necessary tasks, and make sure you give context and a relevant problem to solve (use different language than in the UI)
5. Run with 25-30 participants (unmoderated or moderated)

Task script example:

• Find out how to sign up for a subscription
• Find the help center to answer common questions

When I first started doing visual testing more regularly, I was terrified to apply these methods to my toolbox. I felt like I might mess up and give my team invalid results.

However, when I focused on the perception and the correct goal, I found that these approaches can help teams with certain decisions beyond usability testing alone.