How did you pay your last bill?

Ten years ago your answer might’ve been: “I sent a check in the mail.”

Today, it might be: “I got an email about it on my phone, set up an auto-payment on a website, and received a confirmation through a mobile app.”

Increasingly, our users’ experience of our product is omnichannel and distracted.

The digital world is a dynamic space, with a seemingly untrackable amount of potential product entry points. Your users toggle between the products you build and a bevy of other tasks—messages, lists, phone calls, and to-dos.  

This landscape poses a challenge for researchers looking to understand the full context of their users’ digital experience. Increasingly, getting that full-context requires leaving the lab, and leaving behind the constraints of traditional usability testing.

As a research advisor at dscout, I’ve run dozens of remote studies across complex, shifting, digital landscapes. Here are some best practices, and some sample study designs, I’ve found particularly effective for researching and designing across platforms.

New: Omnichannel research on dscout

dscout just released new omnichannel research capabilities—allowing you to run longitudinal studies that participants can complete on both desktop and mobile. Learn more here.

Researching your website (however your users engage with it)

Websites are the first place that customers/users go when they’re looking for information about your company. Though it seems like our computers are often glued to our sides, it can be difficult to see beyond the screen and hear what your customers actually think.

Traditionally, “over-the-shoulder” lab studies were the default mechanism for studying site interaction. But while lab work might get you a baseline understanding—it can often feel sterile. It won’t reveal the nuance you’re looking for if you hope to grasp what’s co-occuring to your users as they navigate your webpages.

In many cases, remote video research will offer you rich data, closer to your user’s actual, real-life, site-browsing, experience.

To better capture these experiences, it’s often helpful to start with an idea of your best case scenario data looks like.

• What are the “media artifacts” that would be most impactful to your stakeholders? Would a screenshot of the tabs users have open be more useful than looking at their “time on page”? Would a video of them explaining their process, their distractions—of them returning to your site lost—make an impact?
• How can participants submit this data? Would a photo of their screen do the trick? Will a desktop screenshot capture the data you need? Are they able to easily record and send video?
• What are you looking to capture in the first place? Why users are there? What are they struggling with? What are their wishes, or ideal outcomes? These may sound like standard “usability test” asks—but getting them answered in-context can add a lot of insight.

Once you’ve determined what your ideal data looks like, design your study so that you don’t stray too far from it. Grounding your participants to your goal, with minimal supplementary pieces of information, better capitalizes on the “fleeting moment” you’re studying. When you ask them to do, or explain, less—you can focus on the context more.

A few worthwhile research ideas for websites:

1. See how your site compares (and if it’s on their radar). Ask participants to share three websites they frequently visit while engaging with a product or service in your space. See if your site pops up naturally within the fray, before asking them to visit and compare you to your competitors.
2. Screen share when you can. If you want to see the end-to-end journey of your participant’s engagement with your site, conduct a remote in-depth interview. Speak with participants both inside and outside your core user base so you can compare potential and target users. Ask them to screen share—messy tabs and all—and give their thoughts as they engage.
3. Screenshot if you can’t. If you’re looking to get a sense for how the mobile and desktop experience work (or don’t work) together—have your users submit screenshots of the same, or corollary, experiences on both.
4. Get scrappy. Want video of a desktop experience? Have your participants film their screen with their phones. Want video of the mobile experience? Try the “computer hug” method. Ask participants to set turn their web cam on, turn their computer around, and film themselves navigating your site on mobile.

Best practices for websites:

• Identify your best case datum; a screenshot, a “think-aloud,” or something else?
• Anchor your participant to that best case and build a few questions around it.

It’s key to get a baseline understanding of your users experience, before you dive straight into task completion.

Katie Masciopinto

Researching for mobile: in-context when you need it most

If you have an app or mobile-based digital experience, remote mobile research can dive beneath the glass. In these cases, it can be helpful to let your customers choose which modality will be the best way for them to show you moments that matter.

For many, a screenshot and selfie-style video will be a great way to bring out the nuances of a mobile moment. For example: ask your participants to submit a video response every time they engage with your app over a week. Keep an eye out for highs and lows.

The ease with which your users can quickly snap a photo and film themselves talking about their experience gives you an understanding that’s as close to the moment as you can get. And it should get you visceral context of where, how, and why they’re interacting with you from this medium.

For other participants, an in-app screen recording may be the best way to suss out the intricacies of their experience. Instead of asking scouts to focus on a specific button or banner, provide a high-level prompt that asks scouts to take a tour of the app or mobile experience. Have them narrate what feels most meaningful to them. This allows you to get a feel for the landscape of your app.

Whether you look at videos responses over time, or screen recordings in the moment, it’s key to get a baseline understanding of your users experience, before you dive straight into task completion.

Prompts like these help you get a grip on which elements of your interface bubble to the surface for your users. From there, you’ll know what questions to ask when conducting future usability tests, or what follow-up questions will hit home during subsequent phases of your study.

Best practices for mobile:

• Consider different data types. Ask for screenshots, screen recordings, and open-ended questions.
• Let your participants take the lead. Give them guardrails, but let them submit feedback in ways that feel most impactful or clear.
• Be patient. In-app screen recording can be a big ask for some participants! Keep your instructions clear and concise—and be prepared to offer technical guidance.

Your customers aren’t thinking of your desktop and mobile experiences as two distinct entities. They’re not thinking of the user interface and content as two disconnected spokes.

Katie Masciopinto

Omnichannel research: the multiple spokes on the digital wheel

Your customers aren’t thinking of your desktop and mobile experiences as two distinct entities. They’re not thinking of the user interface and content as two disconnected spokes.

Qualitative research platforms can act as the hub where all of these interconnected elements of the digital experience can be brought together.

You can take advantage of the agility and speed of remote qualitative platforms to efficiently get research into the field, and to identify the spaces in the digital experience where you should be focusing your attention. From there, you’ll have the ability to launch follow-up studies to focalize your work.

While you can certainly launch multiple studies to dig into the desktop and mobile spaces, the most powerful studies often take these spaces out of their silos by utilizing a flexible project design that focuses on the digital experience as a whole.

Best practices for omnichannel:

• Program closed-ended or open-ended questions immediately after media data. It will help you filter through your more media-heavy data later on.
• Play with modalities, and think creatively. For example: could participants draw their journeys across channels?

Understanding omnichannel with remote qual: 3 sample study designs

A quick disclaimer: We definitely think of omnichannel research through a “dscout” lens—and find that longitudinal, remote studies are particularly useful for gathering a user’s real, in-context experience at scale. The study designs below are particularly intuitive with the dscout platform, but can be conducted with any tools that allow longitudinal, multimedia submissions.

Sample study design 1: The fact-finder

Whether you’re launching a new digital experience, new to the world of qualitative research, or just trying to figure out what questions to ask, this design allows you to quickly glean insights about your digital spaces by letting participants explore and determine what's meaningful to them. I find that this leads to new avenues and makes experiences better.

The key aspect of this design is letting go of ultimate control. You want your participants to do the driving (with guidance from your prompts, of course).

This design uses an inventory approach, where participants "Show us all the blank..." and describe each blank a bit. For example, "Tell us about your social media toolkit. Show us all the platforms you use."

In the first part of the study, ask participants to show you the top three websites they utilize to complete a goal your product helps them complete (i.e. communicate with friends). Have them follow up by showing the top three mobile apps that they use for the same purpose.

Keep the number of questions at a minimum. Lean on your participants to share what feels pertinent to them. When writing fact-finding studies, it can be helpful to show restraint.

Sample questions:

1. What's the name of this app/website?
2. Take a screenshot of it
3. How often do you visit it in a typical day/week? [closed-ended]
4. In a selfie video, describe why you visit it and why you like it.
5. Describe how you learned about this app or website

Next ask participants to submit moments based on a set “trigger” or “anchor”—aka the moment when you want participants take their phone out of their pocket to describe their experience.

Since we tend to remember the beginning and the end of an experience, it’s often helpful to tightly define these triggers to capture your users digital engagement as they happen. Ask yourself: what must be true of this participant’s digital experience in order for them to engage with this study?

Sample trigger:

Tell us every time you experience a particularly positive or negative moment on social media over a 5-day period.

Lastly, you might ask your participants ideate on the future, as it relates to your product or service.

Sample prompt:

Submit a breakup letter sent to a social media company that is no longer meeting your needs. Take a picture of their letter and read it to us in a selfie video.

This type of ideation is a great way to further empower participants to tell us what matters the most to them. When you’re in the initial phases of a research project, this type of exploratory research can be a helpful benchmark to see if you’re on the right track, or if there are whitespaces for change that you weren’t even aware of.

Sample study design 2: The digital hub

This type of design is best if you have multiple spokes in the digital wheel.

Say you want to investigate job seekers' journey, exploring the multitude of entry points where they search and engage digitally.

First, in this case, you might set participants up for success with an introductory question. Searching for a job is a personal thing, Plus, an easy ask at the beginning of a project can help participants feel comfortable and set the stage for strong and detailed responses moving forward.

An intro question will give you a window into participants’ emotions outside of the digital space. How are they feeling about their job search overall?

Next, have participants show all of the digital tools they use to help them complete their task. Make sure to avoid priming as it’ll limit your responses. For example, instead of asking, “What sites do you use while you job search?” ask: “What tools do you use?” You’ll then get a broader collection of websites, mobile apps, email lists, and other resources.

Afterwards, ask participants to submit a moment every time they complete pivotal task. For example: “Submit a video every time you receive or send an email related to your job search.”

(A heads up: if you’re working with sensitive information, it might be best to avoid asking for a screenshot. A selfie-video will go a long way!)

At this point, you should have quite a bit of data to work with, and you can drill down further into specifics. In the example case, the next step is to ask more specific questions about the online job search tool being explored—specifically, asking participants to sign up for a service on a website devoted to helping them find a job. You want to be taken along for the journey within the journey of signing up for a specific service, including the setup process.

Finally, have participants reflect on their experiences throughout the course of the study, and share what progress they had made. By bookending this study with an introductory and reflection activity that weren’t focused on a digital experience—you're able to see to the physical, emotional, and digital lives of these job seekers.

Sample study design 3: The digital deep dive

This design is great to learn more about a specific app. Specifically, it’s useful for gleaning a participants' brand perception, their feelings about the app content, and how easy the app is to use.

Due to the mobile nature of this research, you’ll benefit from leveraging different types of media.

Begin by asking participants to show the set-up process of the app. For example:

• What are your initial impressions of the process?
• How do you feel about it?
• Take a screenshot of the most painful part of the set-up process.
• Describe to us how this moment could have been improved.

Once the app is installed, focus on the moments of engagement over the course of a week. If possible, leverage a screen recording question, prompting participants to think aloud about what they're doing, seeing, and their impressions. Be sure to follow up with closed-ended questions about the area(s) of the app, emotional state, goals, or whatever is of interest to your team.

These follow-ups are imperative! When asking participants to record their screen, it can be helpful to keep in mind that a little goes a long way. A 60-second screen recording will generate a boatload of data during analysis, so it can be helpful to include subsequent multi-select questions to drill down on the features of your mobile app that scouts were interacting with during their screen recording.

Close your study by requesting “hits, misses, and wishes for the future” of this app. Said another way: ask for the highs, the lows, and the things they want most to be improved.

From there, it might be time to go back to the drawing board—but be sure to keep your participants close. After hearing how first-time users were engaging with the tool, iterate on the app design based on their feedback. Once you’re ready to release the associated changes, re-engage with participants in a follow-up study to hear their thoughts on the new design.

Omnichannel research on dscout

dscout Diary has long allowed participants to log answers to various question types (open-ends, closed end, scale, ranking, video and photo, screen-sharing), wherever they are, via their smartphones.

Now those same capabilities are available on desktop—making it easier to understand the multi-modal user experience with a single study.

Some key use cases:

• Shopper journeys, including inspiration, research, price comparison, and purchase options—See which sites are trusted and the pathways folks take from start to finish.
• Concept tests, especially when mobile vs. desktop differences are expected—Does functionality perform better in one channel compared to another? Benchmark it!
• Experience optimization, which helps ensure stable performance across modalities—Maybe a mobile app doesn't need to "do" everything the desktop does?

However it's used, adding an omnichannel frame to experience research creates a more valid design, matching the world most users find themselves: jumping from device-to-device, screen-to-screen, and across modes. Expanding the data capture lens only offers an opportunity for a more complete picture, regardless of the study's goal or aim.

To see it for yourself, sign up here to get access to a shopper journey demo project.

‍

Katie Masciopinto is a lead research advisor at dscout. She loves to bring people’s stories into the research space, and would consider Abraham Lincoln her favorite human to talk about.