Research Design That Spotlights Highs and Lows of Your App Experience

One of the fundamental challenges of understanding people’s “actual” experience of a mobile app is researching without coloring their experience with your presence. How do you see what users are experiencing without changing what they are experiencing by observing them?

dscout, a mobile-first contextual research platform, is purpose-built to address this challenge. WIth remote tools for app research, you can see the real-life moments shared by app users, hear their likes and frustrations, and understand how users actually experience your app.

Your experience in the usability lab, with surveys and with ethnography will serve you well in designing remote app research, but there are a few other considerations that, when blended with your existing design toolkit, will ensure your app data sings. Here are strategies worth considering in your app research design:

Want tips on analysis that complement our design advice? Stream our webinar on analyzing app experience research, featuring UXPA International president Jen Romano-Bergstrom.

Media Question, Your in-Moment Superhero.

A signature benefit of remote contextual research is capturing multimedia from participants. Using media questions, a user can pour their heart out about your app, take a screenshot of a confusing error message, or even record themselves setting up an account for the first time, all in-the-moment and without leaving their mobile device.

dscout offers three different media question types: Photo, video, and recording a scout's mobile phone screen.

• Screen recording allows scout participants to capture how they use your app in real-time, giving you a contextual window into how your app looks and behaves on their devices. You can simultaneously capture audio, adding a tour guide element to the screen recording. How are scouts describing the actions they're taking with your app? Is it commensurate with your team's expectations?
• Leverage a photo when asking scouts to ideate on a would-be feature or app action. For example, a photos app developer might ask scouts to capture moments when they wish an AI photo assistant could help make their images better for social media posting. See what app moments participants deem worth sharing, and quickly collect batches of real-world examples.
• Use video to ask scouts to reflect on app moments, turning their cameras on themselves to create selfie-style diary entries. What were they trying to do with the app in this moment? What do they wish were different? Scouts are most creative via video; capture and consider their expressions, tonality, and nonverbals. Are they excited when using your app? Use a video to find out. All videos captured on dscout are automatically transcribed, making tagging a snap.

Ensure your mission parts contain a media question where appropriate—even if you replace an open-end with a selfie-style video. And there’s no need to settle for just one media question: Pair an app screenshot with a selfie video for maximum context. Ask scouts to show you the app in-moment, and then have them follow-up with a quick "What are you doing in this moment?" video for depth and visuals into your app.

Open-Ends: Rich Texture, But Quickly.

For a qualitative researcher studying app experience remotely, open-ends are the go-to question type. They afford participants the space and flexibility for description, creativity, and imagination, something the punchiest closed-ended question just can't touch. They should, however, be used carefully in app research, when moments of interest can be fleeting. Here are some tips for formulating effective open-ends:

• Set expectations with a preview statement: If you want several sentences of description in a scout's answer, state that in the prompt: "In a few sentences, describe why you feel that way." Similarly, when a shorter response will suffice, provide that direction: "Give this moment a Tweet-style title." Your scouts will appreciate the guidance, and your data will be better for it.
• Choose your open-ends with care: Running app studies on dscout allows you to be right in the moment with a scout as they're interacting with your app! To maximize this benefit, use open-ends that add context to the app interactions. Avoid creating more than three open-ended questions for a single moment, as showing participants a long list will suppress effort and creativity. Remember, video questions are perfect for capturing imaginative and detailed responses! Scouts' fingers will fatigue if they're shown a battery of open-ends, so choose wisely.
• Redefine an open-end's job: Play with the structure, form, and response universes of your open-ends, just like you would an interview guide: "Encapsulate this moment with an emoji." "In three adjectives, describe this step." "Give this moment a title." You're still tapping into scouts' creativity, but reframing sentence structure to gather richer data for analysis.

Open-ends are such a go-to tool for experience researchers that we often don't consider their impact on our analysis and what we can learn from participants in a timely manner. For app analysis conducted remotely, choose these questions wisely and creatively. If you can't answer "Why are we asking this?", it's worth cutting.

Don't Forget Closed-Ended Questions!

Even for the most qualitatively-minded experience researcher, quant questions are a vital tool. Adding a few closed-ended questions can help inject rigor into your outputs, and it has the secret benefit of scouts tagging your data themselves. Let's explore what I mean:

• First, closed-ended questions surface quantitative indicators to support your qualitative trends. Questions like "How often do you experience this issue?" or "How frustrating was this moment, from 0 to 10?" are easy partners for your user stories and quotes.
• Second, use closed-ended questions to ease scouts' thumbs (they are submitting moments from their smartphones!). Questions focused on features or parts of your app, or steps in a process, are ripe for conversion into a closed-ended list. Providing choices also can reduce the mental burden on participants, safeguarding against quality degradation as a mission progresses.
• Third, and most importantly, a few closed-ended questions peppered into your mission part allows for quick surfacing of the most important moments to you and your stakeholders. Curious about the perceived utility of your app’s "Account Settings" page? Create a closed-ended question defining what page a user is on, and filter to moments focusing on that exact page. Interested in the most painful moments with your app? Create a question asking participants to rate in an entry how confused or satisfied they are, then filter to the most disappointing (or delightful) moments.

Using closed-ends allows for scout-enabled tagging, saving you time and freeing you to focus on the meatier, thematic tagging structures. If a question's responses are potentially finite, make it closed-ended to narrow your scope.

Back-to-Basics: Triggers + Active Language!

More foundational to designing remote contextual research broadly, it bears repeating that for any remote app study to succeed, scouts need triggers and active language. Triggers are your research cues, priming participants to know when a moment is right for capture (e.g., "Any time you're having trouble with our app, show us...!"), and are imperative to getting the right data for your goals and needs. Active language is the principle of grounding your questions in the moment, as opposed to a recall-style (e.g., "What are you trying to accomplish when you do X?" "Rate the ease of ___ from 1-10." "Capture a screen recording of this bug."). You want data as close to the action as possible, and using active language aids in that effort.

These four strategies, combined with the foundations of effective remote contextual design, will ensure your project both captures the data you need and makes analysis largely self-evident. Tags will flow naturally from open-ends; filtering is seamless with a combination of closed-ends; and with a carefully selected media question, you have can see actual app usage like never before.

For advice on analysis of your app research projects, we turned to UXPA International president Jen Romano-Bergstrom for a free streaming webinar on analyzing app experience. Jen’s an experienced remote researcher and dscout user who developed tips and strategies for lightning-quick analysis while studying Instagram’s app during her time at Facebook. With both effective design and analysis, you'll be iterating on and improving your app in no time!