November 17, 2021
November 17, 2021
After years of solely focusing on qualitative research, I finally got on the survey bandwagon and it felt like a whole new realm of possibilities. I could now create questions to ask a large sample of users about their behaviors and feelings and follow up on qualitative studies.
After polishing up my survey-writing skills and sending out numerous projects, my teams were happy to finally get some quantitative data to pair with all of our qualitative insights. Questions about sample size came up less frequently, and I felt more confident in my research presentations.
However, there was still an area I was struggling in. I continued to conduct usability tests, ensuring that I recruited 5-7 participants per segment, but it didn't seem like enough. I couldn't give my team the same confident answers, and I had difficulty quantifying my results unless I recruited a lot more participants.
After a while, I started creating survey questions to ask participants how they felt about prototypes or our product, but I could never find the correct wording. All of my attempts felt wishy-washy and too qualitative.
Around that time, I happened to be attending a meet-up in New York City where one of the presenters spoke about user research metrics. I was hooked and immediately went home and Googled all the ways I could apply them to my work.
From then on, I never looked back.
There are many different types of metrics when it comes to user research. Incorporating quantitative data comes in many shapes and sizes, from product analytics to survey data.
But it wasn’t until the meet-up that I learned about metrics that focused on the interaction between users and the product. These were the missing key to my presentations about usability tests and allowed me to go beyond my small sample sizes.
Here are the two metrics I now utilize the most in my research studies:
The single-ease questionnaire is a one-question "survey" you can ask at the end of each task during a usability test. The SEQ is worded and labeled as:
"Overall, how difficult or easy did you find this task?" OR "Overall, this task was..."
1 = Very difficult
7 = Very easy
You use the SEQ during usability testing. After a participant completes a usability task, you immediately administer the SEQ. Optionally, you can ask the participant why they rated it the way they did, but it isn't necessary. You repeat this with all the tasks in the usability test.
Once you have conducted the test, you look across all the tasks to see the weight users gave to the different tasks. You can do this task by task to find areas of improvement. For example:
Task one: Update your password
User 1
User 2
User 3
Through this, we can see that this task felt difficult for users because they could not locate the key areas needed to achieve this task. With this knowledge, you can go back to your team to improve the experience.
As with any metric, we need to keep in mind some variables:
Unlike the SEQ, which looks at task-related usability, the SUS looks across the entire test or experience. What that means is, instead of looking at each task, the SUS gives you the big picture understanding of the participant's overall impression of usability and experience.
The SUS has ten questions, using a scale of 1-5; 1 = strongly disagree, 5 = strongly agree:
You can use the SUS in two ways:
For example, let's say you ran a usability test. After that usability test, you would administer the SUS to each participant. Once you have completed the usability test, you will find your SUS score:
This could look like:
Your SUS score is 45
The other option is to use the SUS more passively, as a pop-up, to assess the overall experience. I have done this in the past and have received some data through this method. However, I will mention that it is not as reliable as the other approach.
For instance, some people could come to your website, see the SUS and respond blindly without using the website, or they could have only used a portion of the website. Therefore, instead of a pop-up, I have used the SUS at the end of a flow, such as after check-out.
As with any metric, we need to keep in mind some variables:
Although the SUS and SEQ are my favorite metrics, there are others you can try:
I have used all these metrics and have found the SEQ and SUS to be the best performing. However, all of these metrics have been validated across many studies. So instead of trying to pick the best, objective wording, using these metrics helps ensure your data will be valid and reliable.
As with any survey data, this information is self-reported, so it is helpful to use other methods to help qualify the data. For instance, people may perceive a task to be easy, but product analytics may show a large number of people dropping off or failing to click-through to the next step, signifying a problem with the flow.
Additionally, gathering low levels of satisfaction does not tell you where the issues are. Instead, combine a metric like the SUS with a follow-up question about why users gave a particular score or use 1x1 interviews for diving deeper.
As always, triangulating your data is the most effective way to get valid and reliable insights.
Nikki Anderson-Stanier is the founder of User Research Academy and a qualitative researcher with 9 years in the field. She loves solving human problems and petting all the dogs.
To get even more UXR nuggets, check out her user research membership, follow her on LinkedIn, or subscribe to her Substack.