Managing Research Timeline Expectations: When Scrappy Turns Crappy

This conversation is a short excerpt from a more in-depth panel conversation (releasing Jan 13). For more insights from Michele Ronsen, Janice Wong, Noam Segal, and Cristen Torrey, sign up to stream the discussion here.

Defining the subjective

Words matter, especially in the evolving and maturing user-centered research world. We began our conversation with an exercise in operationalization: that is, how we define the phrases that have recently come to describe our practice. "Scrappy" seems to be the most popular of the time-and-speed related terms used to say, essentially, "Do it faster!" How might we define this?

Michele Ronsen | Founder @ Curiosity Tank: "'Scrappy' is definitely cultural, it's environmental, and it's exponential. It's really based on the collective experiences and the interpretation of the experiences. In my experience as a researcher, scrappy would mean, to me, something that is extremely lightweight, focused—and requires a lot of collaboration, agreement, and participation from my stakeholders in order to be successful."

Janice Wong | Senior UXR @ Amazon UX Lab: "...it's interesting to think about how the word scrappy has come to mean different things as our practice of design research has matured over time. For me, it's a loaded word because it's so vivid. It definitely makes you think of specific things. It makes me think of quick and dirty; it makes me think of what was left off the table. It makes me think 'There a trade-off that was made, but we're moving forward anyways.'"

Noam Segal | Director of User Research @ Wealthfront: "I have so much baggage with the term 'scrappy...I recall coming out to the academia [and] the main concern about academics is that they do not know how to be scrappy, or nimble. It's interesting how academia and the industry view scrappy differently. I think they're both getting it wrong to a certain extent. One of my main concerns around this is that I'm not actually sure that we do a great job measuring or understanding what is scrappy at a more tactical level. And so I'm never really too sure when I'm conducting a project whether it's closer to scrappy or closer to crappy.

When a 2x2 muddies the water

Whether "scrappy" is taken to simply mean "faster" or something more nuanced like "iterative," our panels suggests that UXRs consider adapting a plan of action for expecting and speaking to questions about how one's research is (or is not) "scrappy." Specifically, our panelists lingered on the (false) dichotomy of speed and quality, focusing instead on complexities such as project type, stakeholder position, and goal of the insights.

Janice: "I think that it's catchy and it's simple for us to think about things in terms of quality and speed. So it makes sense that [this 2x2] is what we start with. When I was at school, in addition to my design and business coursework, I also took a couple classes on negotiation. That feeds into the way I navigate with my stakeholders, I guess. It's more about growing the pie instead of fighting over a small pie where the only pieces that you have to play with are quality and speed."

Cristen Torrey | Director of Research @ Figma: "The thing that this sort of reminds me of is sometimes researchers will be struggling to figure out how to advocate for themselves—for the space and time to do what they call foundational or strategic research. And there's this push and pull: "I can't get the time to do this." And one of the things that I always encourage people to do—rather than to just focus all of your energy on advocating for the time you think you need for this work— is to start adding 15 minutes of your foundational questions to every study that you do. I've done this in the past. It's great success."

Noam: "I don't really know how to dial up or down the quality of my research. I'm not sure how I would actually do that. But I do know how to dial up or down other things—like the level of confidence I have, the sort of answer the question at hand, or the level of completeness that I feel have in terms of covering the entire universe, the scope of the question."

Michele: "...when I'm teaching new practitioners, or even working with a new team, one of the first questions I'm asking is, 'Are you looking for a smoke signal or are you looking to make a hard decision?' Because that will often help inform what plan is built, how much time we need, what resources to consider. And I use that term smoke signals a lot, and I try to get those smoke signals similarly."

Reframing toward confidence, rigor, & outcomes

A major benefit of strategically pushing back on the need for speed (I had to) is that business decisions based on insights ought to improve, making things better for the end user. UXRs are, after all, researchers, folks concerned about and focused on the rigor or strength of their work...few like rework, even on an unlimited timeline. Our panelists discussed a few ways of educating stakeholders and advocating for rigor over rapidity.

Janice: "...usually, the scenario I find myself in is engaging with a partner where the concept is still pretty early-stage. So there's lots of really juicy exploratory questions that could be asked, but they're looking at it more specifically. So maybe if a technical person has a specific question about a feature, I might talk about how that bubbles up into the mental model that the user needs to have in order to be able to work in that moment and do that specific thing. Then it very quickly kind of feeds into a conversation around methods."

Noam: "If we choose the wrong technical solution and it happens to be the one that costs the most, that takes six additional spins of work. And maybe we're taking some back-end engineers off of another project just to be able to deliver this project in time, that's a huge risk for for the business. So being able to speak that language, understand how the sprints will be constructed, understand the amount of work, understand what type of engineering work would be required, and how novel that direction is for the engineers in that team and the company as a whole. All of these things are really crucial in deciding how deeply to dive into the research when working with technical team specifically."

Christine: "The places where I usually encounter timeline issues are when people want to do something quite big or ambitious and they want to start next week. They have engineering ready to go or something like that, and it's net new or something like that. And you're like, 'Well, I would love a little more time to try something weird or do something a little out of the box instead of thinking, 'Okay. What's the easiest possible answer I can get you on this?'"

Michele: "...many stakeholders are asking for things that aren't possible, if you will. I have found [that stakeholders] have not seen behind the curtains of the process. So it's really about demystifying what actually takes place and what their involvement, and their agreement, and their investment in time means to the process. Showing how that betters the outcome for everyone really goes a long way. Baking that collaboration in from the start when we're identifying assumptions, when we're building that plan, generating that interest in mind, really demystifying it...Once they've been through the process once or twice, it's like they've had a religious experience and nothing can really replace that."

The "new normal" of a mostly remote workforce

Hovering over our discussion was the milieu in which most UX research is (and will likely be) taking place: distributed and remotely. How does distance affect not only the practice of "doing" scrappy research, but managing expectations with those who are asking for it, when a lack of co-presence physically may hinder empathy, clarity of communication, and operations.

Janice: "...it's almost like we have this new bucket of COVID complexities that serves as further focus on why we should be specific about the scope of focus for a project or how we're going to do something. And so now it's almost like everybody understands that things will take longer, and it becomes easier for me to make the argument. That's why I have this long list of guardrails for things that I'm not going to be able to touch on this research study. And why it's even more important for us to take an iterative approach and focus on this to start."

Noam: "I think one thing quite often people who are trying to break into UX research or early career researchers maybe don't fully realize is how pivotal collaboration, and relationships, and partnerships are to the success of research. I get asked much more often about tools and methods and not how to build better relationships or communication. What has changed is our ability to collaborate and partner and build relationships. And we're working very hard to keep up the incredible level of collaboration and partnership that we had before COVID. And it's a work in progress. We've made a lot of advancements and good steps in the right direction, but it's tough in my opinion. It's tough."

Cristen: "I think the only thing that feels like it's a little bit different we've made more investments in asking [participants] to do pre-work, making types of activities, because a lot of times remote conversations tend to take a little bit more time or you're just trying to get into the meat of it a little bit quicker. It's a little less awkward to collaborate on something where you're both manipulating or kind of communicate visually together. We have done a lot of fun things."

Michele: "...many people think about doing research as a series of steps to be followed with some creativity in there, but there is a huge opportunity to educate the newer generation about how important it is to be likable, available, flexible, agile, friendly, educational in the whole collaborative aspect for the sheer reason because most people originally think of research as a roadblock. It's either going to cost too much or it's going to slow us down. So we can enable and remove those roadblocks. So that collaboration and partnership if I could take out a billboard, I would."

This is just a slice of our conversation. We delve into immediate next steps any UXR or insights professional can take to advocate for rigor tomorrow, more issues with a 2x2, and best practices for aligning research goals with timing expectations. Check out the full streaming panel conversation on People Nerds here.