A Comprehensive Guide to Accessible User Research: Part 3 – Conducting and Reporting

Including people with disabilities in user research studies ensures that products are desirable, usable, and accessible for people who are often excluded from participating in the advancement of digital products. It’s a fundamental aspect of inclusive design and ensures that people with disabilities can maintain their independence. This article looks at the considerations for conducting or moderating accessible user research studies and reporting the results.

This article is the last of three:

• Article 1: Planning for accessible user research
• Article 2: Preparing and recruitment for accessible user research
• Article 3: Conducting and reporting accessible user research

This article is for people who have research preparations and recruitment underway and want to learn about the considerations for moderating and reporting user research sessions that include people with disabilities. If you are in the early planning and preparation stages, read part 1, and part 2 first.

Note: in this article, the author is using person-first language, which puts the person before a diagnosis (i.e. "person with a disability" instead of "the disabled"). In this way, the person is the subject, and the disability is a secondary attribute. The author recognizes that this can differ regionally, and depending on the disability being described, and that there are varying opinions and critics regarding this language.

Disability etiquette

Prior to your sessions, ensure that you are familiar with disability etiquette. Disability etiquette guides your behavior so that you are interacting and communicating with people in respectful ways. A basic understanding of disability etiquette can inform your approach to facilitation and moderation. It will also ensure that your participant feels relaxed and comfortable.

Here are some useful tips:

• Don’t use victim language: victim language includes phrases like “suffering with a disability” or “bound to a wheelchair.” Instead use neutral phrases like “living with a disability” or “using a wheelchair.” Be aware of outdated terms like “handicapped.”
• Don’t assume their disability is a tragedy: many people with disabilities are well adjusted to life, and may have strong identities that are connected to their disability. You should not assume that the person is a victim of their disability or that they would like their disability “fixed.”
• Ask before making an accommodation or offering help: always ask if the person needs assistance before providing it. This is true for everyday gestures like opening doors, but also any aids or tools you might use during research facilitation. Making an accommodation before asking can imply that the participant is incapable, or could be seen as insulting.
• Make eye contact and speak to your participant, not their caregiver or interpreter: people with disabilities can often be ignored or avoided. Ignoring the participant by speaking to the interpreter or caregiver can lead to the participant feeling that you don’t view them as equal or capable.
• Take turns speaking: let the participant speak, and wait until they are done speaking before responding. It can be disruptive to interrupt a person, especially if the participant has a speech impairment like a stammer or if the participant is lip reading.

You should also be familiar with etiquette that is specific to the person’s disability or access needs. For example, if your participant uses a mobility aid like a white cane, or walker, you shouldn’t move it from where the participant has placed it without asking first. If a participant has a guide dog, you shouldn’t distract the dog from its job by petting or playing with it.

A collaborative approach

As you start to adjust your facilitation to match the participant’s needs, you may find that there may be times when you need to adapt your facilitation for one participant in a way that you didn’t for other participants. This flexible and collaborative approach may be different for researchers who subscribe to more traditional paradigms of research, where the researcher-participant roles are mutually exclusive, and the protocols are strictly controlled to reduce the risk of bias.

However, in accessibility user research, the researcher should try to find common ground with the participant, seek to understand the participant’s world views, and consider the ethical dilemmas of not providing accommodations or adjustments to the participant.

Location, location, location

Each article in this three-part series has discussed location a lot, and that’s because it’s important to the aims of the research and the comfort of your participants. For this final article, we’ll focus on lab-based research, because it’s unfamiliar territory for the participant, whereas virtual and home-visits are not.

Lab-based research can present unique challenges for accessibility user research. It can be difficult to find a lab that has no physical or structural barriers, and also has access to the assistive technologies that your participants use.

On the day of research, arrive early and do a walkthrough of all the spaces a participant might use before they arrive. Ensure you can provide easy-to-follow instructions from the building entrance to the lab, and other facilities like waiting rooms or washrooms. Determine if signage or way-finding in the building is easy-to-follow, or whether you need to add your own signage.

During your walkthrough, you should also ensure there are no trip-hazards like wires, cables, or rugs on the floor. Being aware of any changes to walkways and halls will also be important to recognize. For example, detecting a ramp could be difficult for a person who is blind or has low vision. If the participant would like guidance, it would be useful for the guide to let the participant know that the floor will change to a ramp.

Finally, ensure that receptionists and staff are aware that you are conducting research, and that they may be required to provide additional support to help the participant find their way to the lab. This is particularly important if the participant has a hidden disability.

Starting the session

After you have guided your participant to the research lab, be sure to guide the participant to their seat. It may not be obvious for them. You might consider using your voice to guide them to their seat, or ask if they’d like you to put their hand on the seat.

Take time to greet the participant and introduce the project the way you would any project. Talk to the participant about anything that might be new or unusual to them, for example, let them know who is observing the session, where the recording equipment is, and when the recording starts.

You might also describe the setting, and reassure the participant that a research lab is basically an office. If there are any unusual noises that are coming from within the room or even beyond, it may be useful to explain where these sounds are coming from. This will help put your participant at ease, and avoid the participant being distracted during the session.

As you would with any research, let the participant know that the purpose of the research is to test how well the website, app or other technology works, and that this isn’t a test of their skill or knowledge. You may need to emphasize this for some participants, especially those who have not participated in research before, or for those who may actually be used to performing tests as part of any rehabilitation or therapy they might be doing.

Allow time for the participant to set themselves up on the test device. This might include adapting the assistive technology software to the settings and preferences they’re accustomed to. If you have a participant using a screen reader, you may need to ask the participant to adjust the speaking rate so that it is suitable for you and for recording. It is common for screen reader users to have the speaking rate set to a very high speed, which can be inaudible for many listeners.

If you are testing with a mobile device, it can be useful to have a phone or tablet stand ready—the participant might use a stand at home and it could be essential equipment for participants with a physical impairment.

The participant may have preference for the lighting as well—ask the participant if they’d like the lights dimmed, or if any other adjustments need to be made before starting. Just make sure that your recordings still work with reduced light.

If the participant is communicating through an interpreter, for example in sign language, be sure the interpreter knows to translate as accurately as possible, and doesn’t offer help or leading questions. While you may need to help your participant at times, the help should come from the researcher and not the interpreter.

During the session

Throughout the research session, use the disability etiquettes mentioned earlier to guide your behavior.

In applied research, or industry research, it can be common for researchers to take notes during the session. Try to avoid this, so that you can give the participant your full attention, as well as observe things like keyboard interaction, or other interactions or feedback from assistive technologies. Taking notes and typing loudly can also distract your participant.

Remember to actively offer breaks to the participant. Take comfort breaks, refill drinks, and spend time away from test devices. This will reduce participant fatigue and ensure high quality data can be captured during the session. If the participant is finding it difficult to continue, you should end the session early and let the participant know they’ll still be compensated for their time.

Be mindful of how your facilitation techniques could be understood by the participant. When researchers are behind schedule, it’s common to say something like, “In the interest of time, I’m going to move us along to the next task.” Using a technique like this to stop a task or end the session may cause a participant to feel frustrated by either not being able to complete a task or lose confidence in their ability to participate in the research. Instead, reword the phrase to “I’ve got everything I need for this task, let’s try something else now.”

You should also be mindful of the research protocols you use. A common protocol would be to ask participants to do a think-aloud, whereby the participants speaks while performing the task stating what they find helpful or confusing about the interface they are using. For a participant with a cognitive impairment, or for a participant using assistive technology simultaneously, the think-aloud protocol could require a lot of cognitive energy. Instead, you might consider doing a retrospective review, and ask participants questions about the interface after they have completed the task.

Finally, keep some pen and paper handy to use as a communication aid. Again, check with the participant first if they would like to use pen and paper before using it. Pen and paper can also be useful for the participant to draw the information they wish to convey, or as a card sort—to help prioritize or organize information.

After the session

Following the session, you need to pay your participant. Check in advance how they prefer to be paid. Online direct deposits into bank accounts are fairly common in research now, and can be a convenient way to pay participants. If you are paying the participant in cash, let the participant know the value of each cash note before you hand it to them. This is particularly important in countries where all cash notes are the same size. If your participant is blind, they might use a physical tool or mobile app to help them identify money, or use strategies such as folds to identify the different values of notes.

If you are doing longitudinal research, or using the same participant many times, you should be mindful of the participant’s current financial situation. If the participant receives social security income, they may invalidate the terms of receiving government income by participating in paid research. Gift vouchers do not necessarily avoid this issue either. It is not your responsibility to decide this for the participant, but it would be courteous to let the participant know. If the involvement in research is a one-off activity, then the research incentive can usually be considered a thank you gift.

Finally, if you need the participant to sign an incentive receipt, ensure that the format is suitable for the participant. You may need to let the participant know where they should sign. A signature guide, which is a card-size plastic or metal cut-out template, may also be useful for some participants.

Reporting the sessions

Following the research sessions, you should spend adequate time with your team to reflect on the research data collected, as well as the research setup. Consider how you might improve your facilitation, whether the location was suitable, and whether the data collected was sufficient. Discuss your participant’s behaviors and find out what the rest of your team found interesting, as you would any research project.

When reporting your findings, it may be beneficial to the reader to split usability issues from accessibility issues. Which are the issues that affect everyone, and which are the issues that affect people with disabilities? This method of categorization won’t always be clear. Some accessibility issues are also usability issues. One way to help with the categorization is to ask whether the issue affects all users equally or whether a person with a disability’s experience is disproportionately worse.

Another method (and again it’s not always clear) is to map the issues back to the Web Content Accessibility Guidelines (WCAG). A limitation of this approach is that you may find there are issues which are clearly accessibility issues, but not covered within the accessibility guidelines. A benefit of this approach is that it can help the team prioritize which issues to fix. Indeed you should provide severity ratings with your findings as another measure to help your teams prioritize the issues to fix.

Finally, be sure to use video highlights from your sessions to playback to the teams. This can be especially powerful, as many people will not have a clear understanding of how people with disabilities use digital products. It may also generate empathy with stakeholders who may still be unsure about what accessibility is really all about.

In summary

This article covered the considerations researchers should take when conducting and reporting on user research that includes people with disabilities. Basic knowledge of disability etiquette and resourceful thinking from researchers will go a long way in providing a comfortable environment for participants. When reporting the issues back to the team, consider what will be most effective for delivering your findings. Oftentimes, it’s useful for teams to see video highlights and understand which issues are accessibility issues and which are usability issues.

Finally, researchers should allow for enough time to socialize their findings. Take what you’ve learned, and share it with relevant teams. Continue to advocate for inclusive user research and share what you’ve learned with your local design, UX, and product communities.

This article is the last of three:

• Article 1: Planning for accessible user research
• Article 2: Preparing and recruitment for accessible user research
• Article 3: Conducting and reporting accessible user research