Research Design Lessons from Uber's Mixed-Method, 7-Market, 100+ Participant Study

In November 2018, Uber launched Uber Pro, a Rewards program aiming to recognize drivers’ quality and commitment.

The beta launch was a complex, multi-market initiative—which required multi-market, multi-method research.

Alongside global discovery research, we conducted a series of mixed-methods evaluative initiatives to ensure Uber Pro’s product-market fit and determine the essentials of the program: a tiering system, the categories of rewards and the mechanisms to access those rewards.

The research approach and scope was as important as the research methods we selected. We had to account for a lot of complex variables: different beta-market sizes (i.e. Chicago is 10 times bigger than Orlando), different driver engagement levels (some drove a few hours a week, some drove six days out of seven), different incentive structures, different levels of competition, and different qualifying criteria. Mixing all of the above, and you could find yourself in a difficult position to isolate the cause and effects of the new program.

After discussing it with Saswati (UXR manager) and Sally (Uber Pro Lead Researcher in the US), we embraced a research program that would be executed simultaneously in seven US markets, and could later be replicated in global markets.

Choosing the right methods

We started by mapping out each research question to the method(s) that would be best suitable to answer it:

• Comprehension and usability of the program required an individual understanding of the participant’s abilities (including tech literacy), background, and product usage. We informed this via a dscout Diary study, home-visits and in-depth interviews. We captured video materials that generated user empathy and provided concrete examples to improve the experience.
• The program onboarding and evolution of the learning process and program engagement was informed via the diary study which lasted a full five weeks.
• Perceived value of the rewards, motivation and attainability of each of the tiers was informed via focus groups with drivers who had a similar engagement with the Uber platform and provided a similar service quality.
• The program awareness, rewards’ preference and the long-term impact was better informed via a self-reported survey, comparing pre-launch and post-launch results.

In each US market, we recruited 4 to 5 participants for each focus group, 7 participants for the diary study. Based on their diary study entries, we selected the most expressive drivers and asked them permission to bring a videography team to their homes to make a mini-documentary. This resulted in a large sample size of over 100 participants for a qualitative research. This was mainly due to the number of segments we had to cover: a matrix of 7 markets and 4 driver segments in each market.

Monitoring engagement through a diary study

Right after the beta launch, we started a 5-week Diary study in dscout to understand the impact of Uber Pro in drivers’ experience, motivation to drive, and sentiment towards Uber.

Why a diary study:

Unlike other research methods, dscout Diary brought us very specific findings about participants’ engagement with the program, motivation to drive, and their comprehension after each of the official communications.

Generally speaking, longitudinal, remote qual offers a few distinctive advantages:

• Sensitizing users with the topic: Diary studies are invaluable for having participants reflect upon their life in-context, and offer up future intent.
• Optimizing our qual efforts: Home visits are expensive. Diary studies allowed us to adapt our discussion guide for follow up with other methods. And they gave us increased confidence that we had our participant sample right.
• Providing real-time insights: Because Diary studies have participants log based on a trigger, you get accurate responses in real-time. Oftentimes, we’d here from them directly from their cars—right after they finished an airport ride or tried a new reward. You don't have to worry about participants forgetting.
• Comparing responses over time: By having users answer the same questions over a period of weeks, we were able to explore and report on feature usage— in-depth and over time. We could gather information about what "regular" use looked like after a feature launch, monitor how a new product worked, and understand user behavior changes from before and after launch.

In short, diary studies allowed us to get deeper level insight of what changed for participants—rather than leading us to rely on just metrics. It showed us our participants’ in-the-moment, unbiased thoughts, and showed their interactions with the product in a variety of settings and circumstances. And made it such that we didn't have to constantly meet with participants in person to understand triggers for change.

How we designed the diary study:

Drivers were prompted to answer three dscout “parts”—completing a repeated series of tasks each week:

Part one focused on planning at the beginning of the week to capture their intentions and other commitments.

Example Question:

• "How many hours do you plan to drive this week?

Part two captured the end of their week. The goal was to understand how their driving behavior matched (or didn't match) the plan and why. This gave us insight into the effects of Uber Pro components and their satisfaction with the rewards they experienced.

Example Questions:

• How many hours did you drive this week?
• How well did that match your plan? Share your answer in a video format.

Part three asked for a deep dive on a specific reward that we called “Your thoughts on…”

Example Questions:

• Have you tried the [feature name] yet? Share your impression of it.
• How valuable do you find it? Tell us why.

A bonus video-question was optional for drivers to answer. We phrased it as: “Did you just have a moment related to the rewards program that you want to share with us? Describe where you are and what happened. Be as specific as you can. The more entries, the better!”

Learnings from our diary study design:

Include an "always on" bonus question

Drivers would share their delight and appreciation right after enjoying an airport priority dispatch. They’d recount their frustration with a trip cancellation at 2am in the night. Or they’d reflect upon getting home after a full day of driving.

A number of factors helped for this bonus question to become so revealing: it was the only contextual, optional, “answer when you want” type of question. This gave drivers flexibility to be themselves and share their videos with palpable emotions, in the context where it happened.

Anticipate some drop-off

Some drivers in the Diary study didn't complete all 5 weeks. We anticipated this possibility and over-recruited to make sure we had drivers from all tiers and markets at the end of the study.

Take a “less is more” approach

There was a lot of pressure to get the program right: the investment volume, the competitive landscape, the number of teams involved, the company working to go public. In light of this, we overdid it. I forgot the principle of “less is more” and went above and beyond to uncover objective insights for each market. In hindsight, we could have limited the duration to three weeks, divided the study into lighter weight studies, and limited the participant numbers to make sure we could effectively analyze and compare their responses.

Design your study for easier analysis

A few things we learned to make analysis easier for such longitudinal, remote studies:

• Including multiple choice questions throughout the diary study helped us have quick results to share along the way.
• Giving complete, accurate instructions on video quality ensures you can use all participant answers from day one.
• Encouraging users to answer questions via the dscout messaging function proved to be effective, insightful, and considerate (drivers sensed we truly cared about their answers).
• Keeping analysis within the platform could have saved us time manually copy-pasting. Consider having an analysis-focused training with all the researchers involved.

A collective approach to analysis:

We had a researcher focusing on each market, conducting and sharing preliminary findings as soon as they gathered the evidence. This setup allowed us to capture video reactions, usage, perception and product bugs, and share them on a broad internal newsletter on a weekly basis.

Once the study closed, we created a findings report template at a market level to standardize and make the comparison across markets easier. After having completed reports for each market, we gathered all of the researchers, data scientists and designers in one room for a one-week research sprint.

Analyzing as a cross-functional team allowed us to triangulate qualitative findings and experiments results to produce very powerful insights. We first created a base level, compared markets and generated insights and concrete opportunities for each of the themes: the user journey, the rewards, the criteria and the communication, comprehension and impact of the program.

Summing it up...

Best practices for designing, executing, and analyzing a diary study:

Study design:

• Clearly define your goals and double check the suitability of a remote tool.
• Each dscout mission (individual research project) is composed of Parts. Each part should be doable in less than 8-10 minutes to keep participants engagement high.
• Limit the number of participants under 6 per cohort and under 20 in total. Too many participants makes the analysis challenging and does not add too much value.
• Ask only the questions that you are willing to analyze. Combine open-ended, video and multiple choice questions. Multiple choice questions are helpful to glance results and share them quickly with stakeholders.
• Work on the invite message to make sure you introduce the goals of the study, the requirements (time required, phone, laptop, etc.) and the compensation.

Execution:

• Define a process to analyze and report early findings.
• Follow-up with participants to clarify or deepen answers by texting them within the dscout platform.
• Leave space for an always-on BONUS question to gather contextual videos right when they happened (i.e. a woman who had to cancel a trip at 2am because she did not have a car seat for babies). It is very powerful to uncover deeper emotions and context of use of your product.

Analysis:

• Try as much as possible to do the analysis within the diary platform to leverage some of the tools and gain efficiency.
• Use tags to more easily sort entries and find themes.
• Rate your participants to help you later if you need a follow-up study.
• Highlight key quotes from answers or video transcripts.
• Create and download video clips or playlists for high-impact shareouts.
• Identify themes by filtering answers based on cohorts, multiple choice answers, or tags
• Review starred clips to add videos and quotes that illustrate your findings.