How T-Mobile Used dscout to Harness Qual Data on Their Team’s (Lean) Timeline

Describing research at T-Mobile requires a lot of adjectives.

The user data they need most comes from studies that are mobile, agile, in-context, longitudinal, proactive, and challenging to do efficiently.

But as their UXRs work to incorporate research into fast-moving, fast-building, ready-to-test design sprints—collecting data efficiently becomes imperative for product success.

We talked with Principal User Experience Researcher Andrea Lindeman about how they’ve worked to do thorough research, with highly specific recruits, on a timeline that makes sense for their stakeholders.

The challenge:

At a company like T-Mobile, mobile research is a necessity, and longitudinal, contextual research is often needed. And executing mobile, longitudinal, in-context research can be a resource drain.

“Oftentimes, we're interested in very concrete moments that happen intermittently,” says Andrea. “For example, understanding the context and experience of someone paying their bill.”

“Shadowing someone in real life until the moment they decide to pay isn't an effective use of our time. And it's way more useful to see the actual moment than to tell users in a lab 'Let's assume you just got your bill...'"

Time and resource constraints can be more of a roadblock when your company moves fast and is hungry for insights. Product teams works at the speed of agile, and research teams had tried a plethora of tactics to keep up: Google design sprints, pre-scheduled in-person usability sessions on a regular cadence, and longitudinal studies that combined manual diary entries over time and wrap-up focus groups.

“Everyone wants to get things out quickly and address problems they’re seeing,” says Andrea. “But on the research side, that hasn’t always translated well to getting ahead of projects, and being able to influence things before they go out.

“It put us at a disadvantage; because of the pace projects move, we were mainly coming at things in a way that’s reactionary, rather than proactive. We ended up testing super granular features, and we weren't able to think more high level about why are we doing what we're doing.”

The solution:

For UXRs at T-Mobile to be successful, they need to pool a large number of specific users, gather their feedback quickly, engage their stakeholders successfully, and give “qual” data “quant-like” credibility.

“When we do in-person lab studies, we encourage stakeholders to observe the sessions. There's nothing more powerful than watching a person totally fail your design. But people don't often do it—schedules don’t often permit,” says Andrea.

“With dscout, designers and product managers can watch the videos at their leisure, when they have free time. And so they do. A three minute video they can watch at their desk is a lower barrier to entry than observing a full session in a lab.”

Beyond accessibility, teams also started to regard research as having more “weight.”

“When you're testing usability of a few different designs and asking 'which design do you prefer,' it's hard to make the case based on a lab session with five to eight people,” Andrea says, “You can’t go back to your stakeholders and say, “well, three of five people liked this one, so let’s go with this one.’”

“When you see that fifteen of twenty preferred one or the other and have rich video data to understand why, that feels more reliable.”

Case 1: Out of the app and in context

When we launched T-Mobile Tuesdays (our rewards program) we had a lot of behavioral analytics data within the associated app. But once people save offers and go to redeem them—they’re out of the application. We lose track of what they're doing, what happens to them, and what their experience is.

dscout allowed us to really effectively track where our users go and helped us to better understand their experience outside the app. They're able to record their screen and upload what they do—versus being constrained to a specific prototype or application—which was so powerful for us. We could also capture users when redeeming offers in a physical retail store to understand the in-store experience as well.

It was the first full picture that we had of customers redeeming various offers. And naturally, we found the experience wasn’t always delightful. So we were able to then go back to our partners and say, "Hey, the experience redeeming your offer isn't as good as it could be." And we were able to then make some changes on our end as well.

Case 2: More effective beta bug zapping

While the T-Mobile app was in its beta period—before a major redesign—we did a dscout study to test it. Participants signed up to interact in the beta over the course of four weeks. We asked them to do some specific tasks, but we also asked them to, whenever they went into the app, record their natural usage. 

If they were going in to check their data allowance, or pay their bill or what not, we asked them upload those moments as well. In this way, we could capture key interactions with the app in users’ own contexts and timing, resulting in much richer and true-to-life insights than we get in the lab.

Anytime a user submitted a video with a serious issue I was able to upload it to our project slack channel and say: “This is a bug.” Watching the videos back, we were able to replicate how the user encountered the problem—and we resolved countless issues that way, in real time, before the new app was launched.

The impact:

Relying on remote research at T-Mobile has been instrumental for expanding scope and capacity. With greater access to user data, stakeholders can move on research more independently and effectively.

"Once, I had finished a study and collected the data right before a week’s vacation,” Andrea says “I shared those videos with the team before I left, and by the time I got back, they had already ingested much of the information and started to act on it without me."

As the stakeholder’s access to data has improved, their capacity for data collection has expanded, the team’s scope has had more room to evolve. The path forward at T-Mobile has begun to look a lot less “reactive” and a lot more “holistic.”

“Our team is being more involved at the strategic level and thinking bigger picture—looking at the user experience end-to-end,” Andrea says. “In the past, things have been developed in silos. I think really our role now is stepping up a level and saying, ‘Okay. We have to think about the entire user experience as a whole and see where the pain points are and how we can address them holistically.”