How to Start a User Research Review Practice (And Why Your Team Should)

More often than not, I hear about how user researchers work in a vacuum—either separate from other researchers or their respective product teams.

While working independently can definitely award you some awesome creative freedom, it can be difficult to bounce ideas off others or get feedback on your work. I always say the best way to learn and grow is to get feedback from others, and it is especially essential in our role as user researchers.

Even when a few of us worked as researchers, we didn't always work together in my previous roles. In fact, more often than not, we wouldn't have time to double-check a methodology or get feedback on an approach since research can move so quickly.

When faced with this problem, I looked toward similar product roles to researchers, which is when I got inspired by design reviews. In these reviews, designers would present work or talk through a prototype before usability testing and get feedback on their approach and flow.

I created a user research review at that organization, and I have implemented it in every role since! Don't have a team of researchers at your organization? You can easily ask designers, product managers, and developers to give you feedback.

Why user research reviews are worth it

The best way we learn is to have open-ended conversations with others on our approaches. When you explain a concept or idea to others, you more easily find opportunities for improvement. You get a diverse set of perspectives on a problem you are struggling with or approach you are unsure about.

When you receive constructive feedback from others, you expand your knowledge and shift how you think about problems. You may learn to look at things slightly differently or include an extra few steps to your approach.

The purpose is for others to provide objective, outside insight and questioning so that you can self-evaluate your work. The feedback provided during the research review is to incite contemplation and discussion. Being able to question your own work is key to being a researcher—we have to be curious about others, ourselves, and our work.

In addition to learning from others, it is also imperative to learn how to give others feedback. This skill is helpful if you are interested in becoming a manager and something that will help you throughout your entire career.

User research reviews will help you work within a growth mindset, which means being open to trying new techniques, knowing it could fail, to learn how you can improve.

How to set a user research review up at your organization

I like to structure my user research reviews as weekly recurring meetings in our calendars, usually about one hour long (so people are more likely to come). Within this format, typically 1-2 researchers can present their work, with each researcher having about thirty minutes to present and then get feedback.

A sign-up sheet gets circulated so people can sign-up for a slot a few weeks in advance. Not everyone should always attend the feedback session—only those who feel they can provide meaningful feedback to the presenters.

Submit work for review

Since research is such a holistic process, with each step impacting the next, it is important to enable researchers to review any stage of the process and whenever they feel they need feedback.

The way we structure our research reviews is to get feedback at these separate stages:

1. Research brief review: Done once a research brief has been created, but before any research has started. Best if stakeholders have aligned on the research, but also okay if reviewed before. A subset of this can also be a discussion guide review, where the researcher can ask for feedback on their discussion guide questions.
2. Research session review: Done after at least one research session has been completed and can occur during or after the project is completed.
3. Analysis review: Done before the data has been analyzed to get ideas for analysis or after the data has been analyzed to retrospect on what was done
4. Presentation & research deliverables review: Done after a presentation deliverable has been drafted or made, but before the research has been presented to stakeholders.
5. Stakeholder management review: Done during or after the project is completed. For this review, the presenter will talk through how they managed or are trying to manage stakeholders during the project.

User Research Review Guidelines

Every feedback meeting should have guidelines to follow to ensure the feedback is as constructive as possible for the presenter. Critiques are focused on the work being presented, and never the researcher who is presenting the work.

All feedback about the work should come in the form of:

• Suggestions
  • Do: "What if you had people go through the entire usability test before asking them questions?"
  • Don't: "You should have had the users go through the entire test before asking them questions."
• "I" statements about objective behaviors you notice
  • Do: "I notice you cut off the participant there before she finished her thought."
  • Don't: "It was rude the way you cut off that participant."
• Open-ended questions that the researcher presenting could reasonably answer in multiple ways
  • Do: "Why did you choose interviews instead of surveys?"
  • Don't: "Don't you think you should have done a survey instead of an interview?"
• Consequences or risks created by decisions
  • Do: "If you only use our current customers, it could limit the generalizability of your insights. What do you think?"
  • Don't: "You need to use non-customers. Otherwise, what you find won't be accurate."

Agenda of the review

We structure each review with a recurring agenda, making it easier for people to present and for observers to know exactly what they are in for. I have even created templates for people in previous roles, so presenters remember to put all the relevant information.

Introduction

The presenting researcher tells all the information they have for the project and then what exact area they are looking for feedback for. This can include:

• Quick background of the project (what and why of the research)
• Research questions the researcher is trying to answer
• Research goals or objectives for the study
• Hypotheses the researcher is trying to test
• The stakeholders involved and any stakeholder struggles/obstacles
• The participants the researcher is thinking of recruiting
• Time constraints (e.g. deadlines)
• Blockers the researcher is coming across
• The priority of the project
• Deliverables or outcomes of the project
• Analysis methodologies

The more information and context the presenting researcher can give the group, the easier it is for the observers to give feedback. In some instances, when a complex topic has been presented, the researcher might send a document for the group to pre-read before the meeting.

Once the introduction is done, the researcher will talk about exactly which area they need feedback on. The researcher who is presenting should take notes on the feedback, but there can also be a designated notetaker for each meeting—we like to rotate this responsibility!

Research plan/brief review

The researcher presenting provides context on any of the following information they have:

• The research questions, goals, objectives
• The chosen (or considered) methods
• How the methods help answer the research question
• The participants to be recruited (including markets, number, criteria)
• How the researcher will recruit users
• The analysis plan
• The final expected deliverable
• A discussion guide, if applicable

Once the presenter has finished, the discussion begins. If the group is having a hard time starting the conversation, some guiding questions can help spark discussion:

• Are the research questions/goals/objectives sound? Can user research answer these questions?
• Are the methods presented the best way to answer the research question?
• Are the methods reasonable given the stakeholders and the timeframe?
• Are there other ways to answer these research questions?
• Are the participants appropriate to answer the research question?
• Do we have the capacity to recruit these participants?
• Are there risks to the participants?
• Does the analysis plan provide answers to the proposed question?
• Is the analysis feasible, given any constraints?
• Are there other ways to analyze the data to answer the proposed research question?
• What risks are inherent to this research plan?
• Does the final deliverable seem appropriate for the stakeholders and the research need?
• What other deliverables could be considered for this project?
• Does the discussion guide ask questions that answer the research goals/objectives/questions?
• Are the questions in the discussion guide appropriate?

Research session (video) review

During the research video review, the researcher plays back a session with a participant, and the observers watch for opportunities for improvement. Before starting, the researcher explains the context of the video in terms of the project's background, research goals/objectives, and can even present a quick discussion guide.

After this, the video starts. Ideally, each video should be no more than 5 minutes long. The researcher can present more than one video, especially if they are having trouble with particular questions in the same study.

The rest of the group watches the video and:

• Call "pause" when they see something interesting
• The researcher presenting says why they think the observer called pause
• The person who called pause calls out what they noticed

*Note: Pause to notice positive things the researcher did as well as ways to improve.

The researcher makes notes of what to:

1. Stop (behaviors they want to avoid)
2. Start (new behaviors they want to start)
3. Continue (good behaviors they want to continue)

Analysis review

The researcher presenting provides context on:

• What the analysis methods were (or will be)
• Any changes made to the analysis from the original research plan, and why
• Other analysis methods being (or were) considered
• The findings

Once the analysis is presented, the discussion can start. Here are some guiding questions if the group gets stuck:

• Was the analysis performed as described?
• What were the stakeholders' roles in the analysis? How does that impact the results?
• Does the analysis make sense for answering the research questions?
• Were assumptions made during analysis that may have skewed results?
• What biases often occur with this type of analysis method?
• Do the insights answer the research question?
• Would other analysis methods better answer the research question?

Presentation & research deliverable review

The researcher presenting provides context on:

• What deliverable they created
• Who the audience is
• The time associated with presenting the deliverables (ex: for multiple stakeholders)
• Plan for how they will share the deliverable across the organization
• Risks for the particular deliverable

*Note: The purpose here is not to run through your presentation with the team. It's to evaluate the deliverable at hand. Keep the conversation at a high level.

Once presented, the group will discuss the presentation and deliverables using some of these guiding questions:

• Is this deliverable appropriate for the stakeholders?
• Are the biases of the research methods and analysis clearly communicated?
• Are the limitations of the insights clearly communicated?
• Is this deliverable appropriate for the timeline?
• Does this deliverable explain the research question and results in appropriate depth?
• Is the research deliverable clear? (ex: will stakeholders understand it)
• Is the research deliverable actionable? (ex: can the team make decisions on this?)
• Would other deliverables help better tell the story of the research?
• Are there other people who would benefit from seeing this research deliverable, outside of the original project stakeholders? Would this affect any additional deliverables you would create?
• How will you activate the findings from this deliverable?

Stakeholder management review

The researcher presenting provides context on:

• The stakeholders involved
• Stakeholder expectations and wants
• Stakeholder level of engagement
• Stakeholders' level of understanding of user research
• Current relationship with stakeholders
• Obstacles or struggles with stakeholders

Once presented, the group will discuss the presentation and deliverables using some of these guiding questions:

• What are the stakeholders asking for? Does this make sense with user research?
• What are some ways to better engage the stakeholders in the user research project?
• How to turn the stakeholders' wants and expectations into a sound research project?
• What are some ways to improve the relationship with the stakeholders?

Wrapping up the review

Looking at their notes, the researcher decides three main things they want to implement differently and three things they want to keep doing in their next attempt at that stage of research.

The research review observers each share one thing they would like to change in their own research work in reaction to the review.