3 Effective Methods for Content Tests (Beyond Usability Testing)

The first time I got asked to test the content of a website, I was stumped. I had no idea how to test information, or words on the page, aside from asking participants what they thought.

Later, I figured I could conduct usability tests and, if users failed the tasks, there was not enough information. That is how I tested content for some time.

While there isn't anything inherently wrong with what I was doing, it usually didn't get the best feedback for my teams. I would present the usability issues we gathered. We could conclude that the information on the website wasn't quite right, but I couldn't provide more direction.

Whenever a content test came up, I felt apprehensive, and my team seemed deflated by the lack of clarity. I couldn't be afraid of content tests forever, so I dug deep into the world of dedicated content testing. Through research and practice, I quickly realized other great ways to test content and get more concrete answers for teams directly.

Jump to:

• Why content test?
• Three methods for content testing
  • Cloze tests
  • Recall-based tests
  • Highlighter tests

• What to consider before diving in...

Why content test?

When users achieve their goals, our organizations win. We accomplish our acquisition, retention, and conversion goals. If our users cannot find what they need, they’re less likely to sign up, purchase something, or stay loyal to a company.

Content testing exposes whether or not users can adequately find and understand critical information; it’s key to bringing these insights to light, and putting debates on copy changes to rest.

Content testing can go beyond explaining whether or not the information in your interface is clear. It can tell you whether the tone of voice makes sense and if users can efficiently process and digest the text. If someone comes to your website and cannot understand the value of your product, or you come off sounding too jargon-y, the likelihood they will purchase starts to drop.

Three methods for content testing

There are three main ways, outside of usability testing, to test content:

Cloze tests

A cloze test is a fancy way of describing what some of us might remember as "Mad Libs." In this test, you take a sample of text, remove specific words, and ask participants to fill in what they believe the missing words are. Through this test, participants must rely on the context and their knowledge of your product.

Cloze tests are great for determining how appropriate and understandable text is for your audience. These tests are beneficial when dealing with highly complex topics, such as legal or healthcare information. However, you can still use a cloze test to assess the understandability of any website.

If you're interested in running a cloze test, follow these steps:

1. Take a bit of text from your website, about 250 words. Use text you aren't sure about, or people have had problems with in the past. For example, if customer support gets many calls on how your product works, test the text currently on your website about that subject.
2. Once you choose the text, take out every fifth word and replace it with a blank space. You will, ideally, have around 25 blanks in your text and no more than 50. Mad Libs can be cognitively exhausting!
3. Ask participants to fill in the blank spaces with the word they believe should be used.
4. To score the test, count the number of correct answers and divide that by the total blank words. Then, turn this number into a percentage. For example, if I had 25 total blanks and a participant got 15 right, they would get a score of 60%.

After scoring, you can use the following benchmarks:

• 60% or higher indicates the text makes sense to the audience.
• 40% - 60% indicates the reader might have a hard time understanding the text.
• Under 40% indicates the text is not well-understood, and people will struggle with processing it.

Recall-based tests

A recall-based test assesses how memorable and comprehensive content was. There are two processes humans go through when reading: decoding, and then comprehension.

Decoding is seeing letters and putting them together to make words. Comprehension is about what the words mean in the context of each other. For instance, have you read a sentence, then got to the end, and had no idea what you just read?

This is why it is essential to differentiate understanding (ex: "Did you understand what you just read) and comprehension (ex: "What does that sentence mean to you?"). Recall-based testing can help with this.

Here are the steps to conducting a recall based test:

1. Choose a text that has been previously confusing to users or new content you are interested in testing.
2. Determine factual questions that will assess comprehension of the text. For example, if you are testing a car dealership website and there is a vital payment plan option, you can ask users to recall that information to see if they could comprehend it.
3. Present the content to the user and give them a chance to read it. Depending on the amount of text, this could take a few seconds to a few minutes. I ask the participant to take as much time reading the text as they would at home (or outside of a testing environment).
4. Once they finish reading the text, ask factual questions about the content. You can ask these follow-up questions verbally or through a survey.

If participants are having a hard time recalling information, it would indicate a rewrite. After the post-test survey, you can go back and ask some open-ended questions about how the participants might improve the content.

Highlighter tests

Finally, one of my favorite ways to test content is the highlighter method. Here, you ask users to indicate what’s clear and unclear by highlighting text. This test can help determine precisely what needs to be rewritten. It can also reduce the amount of text on a page to focus on what is essential to the user. The highlighter test is excellent for determining value proposition and what information is necessary to help users achieve their goals.

Running a highlighter test is relatively simple:

1. Choose content that you want to test. It is okay to try longer passages for the highlighter test as the actions don't require a high cognitive load for participants.
2. Define what the different colors mean. Most commonly, people use:
   1. Green highlighter for "this is useful or helpful information."
   2. Pink or orange highlighter for "this information is confusing or unnecessary."
3. Present the passage to the user and have them read it through without marking up anything. After, explain the colors and allow them to highlight the selection.
4. Save some time after the session to ask any follow-up questions on why something is useful or confusing.
5. To score the test, underline the most frequent green and red passages to present to the team, alongside any qualitative comments explain why.

What to consider before diving in...

First and foremost, define your goals! Sometimes content testing can get confused with concept testing or usability. In content tests, we are zeroing in on the text's comprehension and readability. If you want to test both content and usability, I would recommend separating the projects, if possible. Defining your goals will also help you choose what type of content testing to use.

The most common goals for content testing are:

1. How understandable the text is for a given audience: cloze test, highlighter test
2. The readability of the text: cloze test
3. If users can comprehend the meaning behind content: recall-based test
4. Understand which content is unnecessary or confusing: highlighter test

Once you have defined your goals, you can choose your method. I recommend moderated testing for these approaches. With moderated testing, you can follow up with the user and also read body language cues. For instance, you can see someone hesitate while reading and ask them why.

After defining the approach, make sure you recruit the right people. If you are testing with people who aren't interested in your product/service, you won't get great feedback on the content. You should recruit people who are trying to achieve goals or solve problems with your product/service. You can do this through a screener survey or screening interview. For content tests, I recommend speaking to 10-15 people per segment.

Finally, always remind people that we are not testing them! These types of tests can feel much more like a test than usability sessions or 1:1 interviews. Explain that you are testing the content, not them!

You can determine what users truly understand versus what they say they understand through these content testing methods. This is such an important distinction. While people can claim to understand your text, if they don't truly comprehend it or find it helpful, they might not use your product/service. By constantly testing your content, you will ensure users have the best information to achieve their goals and, ultimately, your organization's goals.