Research that Works in the Wild

Mobile research is remote research. It’s a different beast. Observing from a distance means your participants are out on their own, without you on-site to guide them or provide clarifications. If you usually stick with in-person user testing and interviews, fielding research in the wild might feel like uncharted territory.

On the flip side, being remote gives researchers unique access to authentic thoughts, reactions and behaviors — as they happen and in natural environments. That’s hard to beat that when you want to understand real moments in a real context.

When fielding new kinds of research, we need a new set of skills. Here are six considerations to ensure your research design is successful out in the wild when your baby birds leave the nest.

1. Don’t just screen. Audition!

Traditional recruiting channels use phone screening and surveys to zero-in on your target demographic, but they offer little indication whether your selected respondents can excel at following instructions, using digital tools, or expressing themselves.

For example, when we wanted to recruit energetic, articulate people who wear fitness trackers, we included a video prompt in our screener. In addition to our typical screening questions, we asked applicants to record a selfie-style video where they introduced us to their fitness tracker. We could hear how they presented their thoughts and we saw how they carried themselves, which really gave us a sense for how well each person would do in our research. Plus, we confirmed that each applicant does, indeed, own a snazzy fitness tracker.

2. Don’t go in cold.

Before diving in to achieve your core research objectives, start with a more general warm-up. Ask participants to introduce themselves and to reflect on larger themes relevant to the project.

For example: Before instructing a group of participants to start a week-long diary of all the meals they cook, try to get a sense for who they are. How do they spend their days? Who else do they cook/shop for? Do they have special diet needs?

Telling you about these aspects of their lives will give your participants an easy first task, and it provides you with a foundation for user profiles that can anchor the arc of your findings. Then later, when your participant is cooking quinoa with one hand and Kraft mac and cheese with the other, you have the context to deduce that it’s because she’s a paleo-loving, power-lifting mom with a picky-eater kid.

3. Consider the respondent’s level of experience and awareness

Maybe you spend your days thinking about new innovations in luxury cat toys, the browsing behaviors of urban-tweens, or the future of auto loans. Your recipient, on the other hand, almost certainly does not. (Won’t you be lucky to find the one who does!).

Whether you are a market researcher, designer, or product owner, you undoubtedly have a higher level of expertise — or at least a more specific frame of reference — than most participants. Be aware of this bias so you can design your research questions for your participants — not for yourself.

Be on the lookout for unnatural jargon and framing. For example:

• Research Speak “Over the next week, anytime you have a remote interaction with an organization, tell us whether or not you chose a synchronous or asynchronous channel and explain pain points for each touch.”
• People Speak “Anytime you contact a business over the next week, tell us how you get in touch and then reflect on what could have made it a better experience. We’re interested in how you communicate when at a distance (phone, email, letter, app, social media, website, etc.), so there’s no need to tell us about in-person experiences.”

Put yourself in your participant’s shoes to identify these holes. It also helps to pose your questions to a friend (or anyone whose days aren’t spent thinking about new innovations in luxury cat toys). Gauge their responses, and adjust your language accordingly.

4. Anticipate misinterpretation

Before you launch your research, take it for a test drive. Think about all the things a person could do wrong, all the things they might ask you to spell out in more detail, and all the things that may not make sense in certain situations. Run through your research design to identify all the wrong turns a person could take, then close the gaps.

For example, it may seem cut-and-dried to say, “Show us every time you shop this week,” and then ask participants to take a photo of each store. But, what do you want someone to do if that shopping is online? If you don’t tell them how to handle it, they might skip it entirely, and you’ll never know.

When we’ve asked participants for a video tour of their fridge, and we wanted narrations, we made sure to state that expectation. Otherwise, we surely would have ended up with some very silent videos.

5. Frame Instructions with constraints

You may recognize a constraint in the example above: “We’re interested in how you communicate when at a distance … so there’s no need to tell us about in-person experiences.”

Why was that necessary?

Framing research is a balancing act: Too broad, and I lose depth. Too narrow, and I hear only what I was expecting. Grounding your design in thoughtful constraints helps participants know what isn’t relevant to a project without affecting their natural behaviors and choices.

To find the right constraints, start by breaking down the research question to pinpoint exactly what you want to see, then think about all the outside factors that stand between you and that observation. The best constraints limit the noise in your data, without stripping the color and context from your insights.

For example, I was studying how people make transit decisions for both their regular and less typical destinations. My research questions focused on digital tools like transit trackers and mapping apps — what transit situations lead people to use digital tools, and when are these tools under-utilized?).

In my research, I asked participants to capture moments when they’re preparing to go somewhere. Here are some ways I could have used constraints — good, bad and ugly:

Good constraint
“Only document the moments when you’re venturing more than a few blocks.”

Here focus participants on what’s relevant by asking them to capture moments that we expect are more likely to require some planning and routing. Low noise.

As a bonus, we expect to still receive some transit moments when people do not use tools — which will be a valuable point of comparison for the product team.

Bad constraint
“Only document the moments when you’re going to an unfamiliar destination.”

By limiting destinations strictly to the unfamiliar, we do focus participants on moments when they’re likely to use a digital tool. No noise.

However, this approach also strips color and context by preventing people from showing us their daily routing and transit decisions to familiar places — something we needed to observe.

Ugly constraint
“Only document the moments when you use an app or website in your decision/planning process”.

Here we get right to the point and eliminate noise by asking participants to show only moments where they use a digital tool to plan their route/transit.

Not only have we primed our participants (by expecting them to use a process), but we lose all visibility into the moments when someone does not choose to use a digital tool.

6. Be present and proactive

Just because you aren’t in the same room as your participants, it doesn’t mean you can’t have an active presence. When you share feedback and encouragement, people become more engaged. Knowing there’s a human out there who is counting on them and cares what they have to say improves the quality of the responses you’ll receive.

Proactively monitor all submissions so you can catch any respondents who might have strayed off-course. Most are happy to redo their entries if you promptly tell them they’re missing the mark. The key here is offering appreciation for their efforts while specifically clarifying how they can improve.

For example: We asked a group of participants to document their morning hygiene routines by taking videos explaining how each step helps them get their day started. When one feisty participant submitted a video of herself rocking out to Beyonce while feeding her fish, I followed up with something like this…

It takes countless hours in the field to run shop-alongs and conduct interviews with just a few people in a few markets. The advantage of remote tools is that you can scale your project to more people in more markets, while cutting back on your field hours.

But there is other work to do to ensure the quality of your results meet your expectations. Take some of the hours you saved and re-invest them in refining your research design to thrive in the wild. When you audition your participants, clear your questions of assumptions, and proactively communicate with your respondents, you’ll be successful when those baby birds leave the nest.