Topics
October 1, 2020
.jpg)
October 1, 2020
"If we don't do this right, we could lose thousands of dollars."
That’s a warning I got during one of my first user research projects.
And during that project, I missed a lot of research project “steps”—partially because I didn’t know how many steps were necessary, and partially because I was afraid of feedback.
I hadn't checked in with my stakeholders to truly understand their problem or expected outcome. I didn't collect any hypotheses or assumptions the team had on the topic. I also didn't write a research plan (nor did I know about research plans). My follow-up wasn't great, and I didn't do a retrospective on the project.
I was new to user research, so it's understandable why I missed so much on my first tries. However, the gaps in my projects were apparent, and caused wasted time and frustrated stakeholders.
I believed I could "learn by doing," and there is no better way to learn fast than being told by a senior stakeholder, "We could lose thousands of dollars on your mistakes."
Through time, I learned the general sequence of and what was included in a user research project. With this knowledge, my approach to research projects changed. I was more productive, my stakeholders were happier, the research goals were more precise, and I felt in control of my projects.
How did I do it? I created a go-to sequence (or life) of a research project.
The life or sequence of a research project consists of the common steps I go through in each research study. I want to stress that this is a typical journey and research won't always happen in such a linear (and clean) fashion. Often, you have to skip straight to recruitment because of timelines or conduct research while recruiting. Or, you may have to go backward. However, ideally, you will go through all of the steps and in a similar order.
We wrote a whole article on what this lifecycle looks like in practice (including timeline estimates from each of the key stages). We can think of a research cycle in 3 parts, with several intermittent steps. To quickly summarize:
Projects can come from anywhere or anyone—C-level or VP-level stakeholders, product managers, marketing, sales, design, engineering, or the research team itself.
The overarching subject you are studying for a project. It’s a very high-level topic you want to understand more about.
This is the time when you get to ask all the questions to stakeholders. Usually, you meet with the one who requested the study and the product manager and designer involved (if they weren't the requesters).
Research statements are what we are trying to learn or understand better about our users. It is best to build these statements after meeting with stakeholders. The research statement will be a guide on what you will cover during the sessions.
Think about the different metrics the project could impact. Some standard metrics are conversion rate, click-thru rate, retention rate, session length, and download rate. You can use these metrics to track the success of the project.
Research goals are the more in-depth areas we want to explore regarding our research statement. They help us better understand the different aspects of our research statement.
Choosing the right method is essential because it allows us to extract the correct information from participants. Think about your research goals and the type of information your team needs from you, and then set off deciding on a method.
Always be clear about the timeline you would expect from the research study. If stakeholders press you for too tight or unrealistic timelines, suggest conducting the research at another time.
Speaking to the right people is arguably one of the most important parts of user research. If you don't get the right participants, your study can suffer considerably.
The discussion guide is essentially a cheat-sheet to use during the research session. It can include detailed questions or just a few bullet points on topics you want to cover.
Send out your screener survey and calendar links to have people sign up for research sessions. I recommend using a tool like Calendly to schedule sessions. If you have a tight timeline, you can overlap recruitment with beginning the study.
During this phase, you are deep in the moderator role and focused on getting through the sessions. I recommend always having relevant stakeholders come to each session to observe and take notes for you.
This is the time to reflect on what just happened during the session and bring many minds together. It is a nugget-sized synthesis session.
During this phase, you are finding patterns across the research with team members. During synthesis workshops, you typically choose activities, such as affinity diagramming and 'How Might We' statements. I talk more about my synthesis process in this article.
During this phase, you can work solo or with other researchers. I review the patterns found during synthesis and build insights and recommendations. Creating insights and recommendations enables the team to make data-driven and more empowered decisions moving forward. This step is where you make the research actionable. Writing insights can be tricky, so check out this article on writing compelling insights.
Outputs usually come in the form of deliverables such as reports, personas, infographics, journey maps, or papers.
The next steps can include action items you plan to implement post research, or following up with data or more studies. These action items are essential to the project because they make the research actionable.
Keep track of how the metrics change (mainly if you conducted usability testing and made significant changes to the UX or UI). The metrics will also help you prove the return on investment (ROI) of user research across the organization.
If more research is needed, it's time to dive in and start the cycle all over again!
Since I already wrote extensively about a generative research project, I will focus on usability testing. I also find that people tend to conduct usability tests more frequently. Let's dive in!
Imagine working as a researcher at an online property company, allowing users to buy, rent, and sell property online. You and the teams you work with focus on the renter's experience on the app.
One day, a project lands on your desk. A product manager wants to create a feature that allows renters to tour an apartment virtually. It appears to be a great initiative and makes a lot of sense. However, as a researcher, you must use your curiosity levels to push past the surface level.
I will now walk you through each step I would take for this research project.
Technically, this already happened. The research request came in from a product manager that I've worked with before. The proposal seems sound, and I happen to know there hasn't been any previous research explicitly on this topic. If I were unsure about previous research, I would go ahead and look through any repositories.
Additionally, I look to see if there are supporting insights from any other studies run before. I know the team, so I also am aware of the different stakeholders I have to include. I want the designer and product manager to participate in all subsequent meetings and the research sessions.
As mentioned, the research subject is the overarching topic of the project. The research subject for this project would be virtual apartment tours. Very high-level, but focused on a particular area.
After looking at the proposal and defining the subject, I send the research requester an intake document. In this document, I ask them to answer key questions such as:
Once I receive the intake document, I schedule a meeting with the product manager and the designer. During this meeting, we will talk through the intake document and any questions I have. If I have many research requests coming in, I want to make sure I prioritize them correctly. To prioritize research requests, I ask myself the following questions:
It is imperative to work on the highest priority research projects. One way to achieve this is to work on research projects that contribute to an OKR or company strategy/goal. As a researcher, you should know the team and company OKRs and overarching goals. If the project contributes to OKRs or goals, it becomes a higher priority.
The team needs resources and talent to act on the insights that come from the research project. If you do this research and create a list of recommendations, can someone on the team make them a reality? If the team doesn't have the resources, I tend to ask them to wait until they have the necessary roles.
If I come across a project that can impact multiple teams, I prioritize it over more localized research. If I can generate insights that impact a broader audience and strategy, I can widen user research's positive impact on an organization.
Sometimes other teams are better equipped to answer particular research questions, such as data analytics or market research. For example, market research usually touches brand perception and communication-related research, while user research is employed when the project could change the actual product. Also, if you are looking at trying to understand a broader or more generalized perception of a feature or concept, data analysis might be more appropriate. You should prioritize research projects whose questions cannot be addressed by other means.
The most frustrating projects I have been a part of are those where I feel like I am merely validating design decisions. When I receive a request to test whether users like or prefer a particular design or experience, I always go back to the stakeholders to understand if there is more behind the proposal. User research does not inform preference. This type of request is incredibly frustrating if nothing will change, regardless of the research results. I always deprioritize these requests and educate the team on the purpose of user research.
There should be hypotheses present in each research request. A list of hypotheses indicates the team (or requester) has considered the research project's potential impact. Assumptions also help me when I determine research goals and write a discussion guide.
I feel like this is most important but usually not regarded highly enough. User research is a mentally exhausting job both from speaking with users and juggling stakeholders. If not, is the team empowered to do basic research?
These questions, coupled with the intake document, make it much easier to prioritize research requests. I send the stakeholders this intake document to fill out, and then I schedule another meeting to review it together. The beginning of research projects can be full of many meetings, but it is essential to make sure the team is aligned.
During this meeting, it would be critical for me to ask about the original request: a feature that allows renters to tour an apartment entirely virtually.
This is the model I follow when creating a research statement:
“We want to better understand how users [think about/make decisions on/interact with] [subject of research/product] in order to [create/improve] [product/website/app/service].”
In this case, I would adopt that model to the request and information I found out about this particular project. The proposal was about a feature that allows renters to tour an apartment entirely virtually. After speaking with the team, I understand the customer problem of viewing apartments without leaving their current home. Since people are so busy, they want to know whether it is worth the time and effort to view the apartment. The hypothesis is that considering an apartment virtually is more manageable and will help people make this decision.
With this information, I would create a research statement that will be our guiding statement throughout this project:
“We want to better understand how users make decisions to view an apartment to improve the experience of viewing the right and appropriate apartments.”
This statement does not target the exact feature because we don't want to jump straight into solutions. First, we want to understand how people make this decision and then develop ideas that aim to eliminate pain points. We will also test a small concept of our virtual video idea at the end of the interview to gauge participant's reactions.
If you aren't familiar with the organization's strategy, metrics and KPIs can be tricky. For this research project, I would talk in-depth with the product manager about what we could use to measure this project's success. I would also involve product analysts in this process.
We aim to improve the experience of viewing apartments so users can waste less time. I try to use HEART metrics so that I am covering as much as possible. To think of metrics, I ask myself, "What types of metrics would mean we are successful at the end of this project?" In this case, since I am looking at more discovery-based research, I wouldn't assign any hard metrics. If we were looking to usability test the virtual apartment tour prototype, I would define the following:
As mentioned, there are five main goals present in the majority of research studies. I develop research goals by thinking to myself, "what information do we want to know at the end of the study?" With that, I pose 3-4 different goals that would help the team get that information. In this case, I would use the following objectives:
These goals allow us to get at the participant's mental models, which relate directly to the research statement. By answering these goals, we will be able to paint a picture of the current process people go through and how we might improve that process. The last goal is for testing the prototype concept of virtual apartment viewing.
Now, the fun part! There are so many methodologies to choose from, but I always start by looking at my goals. I ask myself, "what methods would get me the answers to these goals?" Since we are trying to understand people's thought processes and mental models, a great start would be 1x1 interviews.
During these interviews, I would ask in-depth questions on how they have gone through this experience. By having conversations in a 1x1 interview, I know I will gather insights that will relate to my goals. I would run a short concept test to understand what people think of the virtual viewing prototype. The sessions would mix 1x1 interviews and concept testing (otherwise known as hybrid research).
I would want the sessions to run about ninety minutes, and they would be held remotely. Ninety minutes can feel long, but we have quite a few goals and need time to get deeper insights into the participant's thought process. We also need time to show and prove about the concept.
After I have a clear understanding of the goals and methodology, I can create a rough timeline in my head. Again, I use the following guidelines:
For this project, I would estimate about 5 weeks. We want to have a conversation with users and show them a prototype test; there would be a fair amount of data. I would want to ensure we have plenty of time to synthesize the information into helpful recommendations. My breakdown would look like:
I like to give the team extra buffer time, just in case something happens. It is always better to deliver something early rather than late.
For this study, I would have a few different questions to ask the stakeholder:
In terms of this study, I would recommend the following recruitment criteria:
A small side note on markets: If you work at a global company, you will have to think about your segments. I generally ask the product manager which market the potential feature will roll out to, and which markets produce the most revenue. This information helps me prioritize the markets to speak with.
During this section, I also touch on the screener survey criteria. Screener surveys help us get the best possible participants for the study. I would recommend:
Finally, we need to decide how we will recruit and incentives.
For this project, I would send out an email to recruit (see the templates below). I tend to compensate users with their time through Amazon vouchers, as they are relatively universal. There are a few other ways to compensate if your budget is tight:
Just as a note, think about your participants when determining incentives! For instance, if you are interviewing struggling, small business owners, don't send an Amazon voucher. Think about what your participants would need and use.
For this project, we would offer one-month free access to the premium version of our platform.
As I mentioned before, I don't write long discussion guides. Instead, I put open-ended questions and general topics. I would include the following in my discussion guide:
*As a note, I create all the above in a shared google doc, so the team is regularly collaborating and aligned. Also, I don't always follow every single step linearly. Sometimes I think about the recruitment before the timeline or methodology before goals. Do what feels most comfortable for you!
I send out the screener survey via email and pop-up. I have my templates for recruiting emails up and ready to respond as quickly as possible. I always have back-ups ready, just in case someone cancels or does not show up.
I've started my favorite part! During this part, I booked my entire week off to ensure I am as present as possible during the interviews. During the interviews, I won't change the interview guide after the first two participants (or not at all, if possible). Any changes or additions to the research project will need to take place in a separate research study.
I would schedule thirty minutes to debrief with the relevant stakeholders after each interview. My calendar invite looks like:
During this recap, I would invite the stakeholders to a Miro board to work together on recapping the information we just heard. I generally facilitate these sessions, but I also participate in them. During my recap sessions, I tackle:
After every seven interviews, I will hold a more extensive synthesis session. This session is a more extended version of the recap sessions. For synthesis in this project, I would use affinity diagrams and 'How Might We' statements. Affinity diagrams would help the team organize all of the qualitative research into patterns and trends. I would likely use the following groups for affinity diagrams:
I would have a separate affinity diagram for the concept test to keep the findings separate. In this affinity diagram, I would use the following groups:
After affinity diagramming, I would use 'How Might We' statements to generate ideas. These are statements that reframe our insights into questions to turn them into opportunities for solution ideas. We use these because they suggest many possible solutions and open the door to creativity.
For example, one of the most significant pain points we found in the research is that some apartments in great locations with reasonable prices had few photos, so people can't gauge the apartment's feel. In this case, we would create the following 'How Might We' statement: how might we help users know which apartments may be right for them before they visit?
After we synthesize the information as a team, I take time to analyze the findings. I typically do this alone or with another researcher (if there is one on the team). This process is when I write insights for the team. The way I define an insight is:
For this project, this would look like:
A. Discovery about human behavior, and the underlying motivations behind that behavior
Does what you found give you a new understanding of users' attitudes, behaviors, or context (inside and outside your product/service)?
Example: People who are serious about moving will visit at least five apartments in one month to see what is available and what they could be missing (behavior and motivation)
B. Information that challenges what we believe about users and how they exist in the world
Does what you found negate or change the way you have viewed users in the past?
Example: Although we believed users wanted to see apartments right after they found them online, we uncovered that users wish to have a virtual tour of the apartment before setting up an in-person appointment
C. The knowledge that reveals fundamental principles that drive us towards seeing users in a new way
Does what you found help you understand the user's mental models on how the world should work?
Example: Users believe real estate agents have essential and unknown information about apartments new to the market, which is impossible for them to get. Without a real estate agent, you aren't getting up-to-date information about new apartments. And real estate agents use this information to their advantage to charge a higher commission.
I include six different areas in each insight:
Put the person reading the insight into the situation. Explain the current situation for the participant, which will also give meaning to the research project.
Example: When we finally decided to move, we were excited about the prospect of finding a new apartment and creating a new home. Searching for flats online started as a fun and exciting experience.
Based on the current context, what was the key learning you gleaned? The critical learning may be an unexpected attitude or behavior. It could also be a problem or barrier your participant experienced.
Example: However, we soon realized that many of the apartment postings had horrible pictures or very few of them, so it was almost impossible to get a feel for how the apartment looked. What was once a fun activity became too frustrating for us.
Explain why a particular behavior or attitude is coming up in the research, or why a participant faces a specific problem.
Example: We then felt we had to see all these places in person, which was a massive waste of time. We couldn't necessarily make a judgment about an apartment in a beautiful area with terrible photos. However, we couldn't spend all of our time driving to see a million apartments. We both work full-time, so we didn't have the luxury of viewing apartments during the day.
Being able to explain the rationale behind why the learning occurred is what makes an insight great, and is the most critical part of figuring out how to help users achieve their goals. Find the frustration that surrounds any given experience, and you will locate the core motivating factors.
We felt uncomfortable ruling out apartments that may have had potential. It made us feel guilty about ignoring a listing that could be our new home. We ended up fighting a lot over this, and the whole situation became increasingly stressful. We didn't want to miss out, but we didn't want to waste time. It was a bit of FOMO, I guess.
What does this particular insight lead to, or what impact does it have on your product/service? Explain what will happen if you don't act on this insight. What does the user feel is the ideal end-state.
We wish there were a way to have a virtual tour of the apartments. We ended up leaving the online platform and finding a real estate agent who could vet the apartments for us. They had a better idea of what each looked like and saved us a lot of time, although it sucked to control someone else. We wanted to do this experience together and enjoy it. And paying that commission sucked. However, in the end, it was more practical for us than the online platform.
The following steps don't mean you tell people what the solution is. Instead, you can write down a problem statement that synthesizes the insight into a concise sentence (or two). The template I will follow is: I am (persona/role) trying to (do X) but (barrier/problem) because (x), which makes me feel (emotion). Also, make sure you give ownership to your colleagues.
As someone searching for an apartment, I try to rule out apartments online not to have to waste time visiting apartments I don't like. However, I cannot properly view the apartments online due to limited or terrible photos, which lowers the trust and confidence in the online platform and makes me frustrated. Ultimately, this drives me to leave the platform and find another solution.
For this particular study, I would create and share two different deliverables:
Primarily for the team, this report would be a succinct overview of the findings and recommendations. It would also include infographics and videos of the sessions. I structure my reports with:
You can take a look at some of my infographic templates here.
The customer journey map would focus on the process people go through when deciding to view an apartment. This map would help the team understand the steps people take during this crucial time. In this map, I would include:
The following steps for this project would be:
Since there were no hard metrics set for the discovery and concept test, there isn't much to monitor. However, I would assign metrics to the follow-up usability tests and continue to watch those as we rolled out the feature.
In this case, it sounds like the virtual viewing is a great idea, and there are insights the team needs to follow-up on. I would recommend following up with usability tests based on the ideas generated from How Might We statements. Let's assume we had three ideas come from the synthesis session, and we voted on the top two. In the following research, I would recommend usability testing those two ideas. For that, I would start this process all over again.
Phew, it is a long process! The above is how I would tackle this research if it were to land on my desk. There is a variation of ways to tackle research projects, so please find what feels most comfortable for you!
Nikki Anderson-Stanier is the founder of User Research Academy and a qualitative researcher with 9 years in the field. She loves solving human problems and petting all the dogs.
To get even more UXR nuggets, check out her user research membership, follow her on LinkedIn, or subscribe to her Substack.
