Fighting for Research Integrity

When is research successful? This is a question we think about frequently on dscout’s User Success team. We spend our days helping dscout customers with everything from research design to analysis. We utilize user feedback to champion product improvements that make dscout better. But, in order to live up to our ambitious team name, we also work hard to understand the ecosystems our clients work in, and the factors that influence their success.

Recently, we convened a group of People Nerds to explore the concept of research success, the roadblocks that prevent us from getting there, and what’s really at stake when challenges arise. In this multi-part series, we’ll share tips from experts and stories from users.

One of the biggest concerns we heard about was the fight for research integrity. Especially when faced with shrinking timelines, work models that leave researchers feeling isolated, and stakeholders that try to exert undue influence. Here, a frank discussion of the struggles—and solutions—related to researching “the right way.”

Speed over quality

Fast timelines, tight resources and competing priorities are just some of the factors that force researchers to make tradeoffs that impact research quality. In part, the popularity of research may be to blame. In some organizations, especially those where research has become a mandatory component of the design process, a “check the box” mentality has developed. And when faced with increasing pressure to deliver insights quickly, it’s not uncommon to resolve that some research is better than none at all.

“There has been such a big shift toward faster, scrappy, iterative projects, but the tools and vehicles that we use to do those studies tend to be very cognitive and analytical. So I find that I’m spending the majority of my time in analytical mode. The challenge is that I think great research comes from a combination of being analytical and intuitive, cognitive and emotional. But the vehicles for fast turnaround research often don’t support all of the modalities of thought that researchers are inherently good at.”
— Alyson Madrigan, Director of User Research, SoFi

“The business world has forced research to move so far away from traditional ethnographic research methods. The result is that it becomes harder and harder to do longitudinal research or true observational research. It’s just not compatible with the need for more data points and the urgent, faster pace.”
— Katherine Lee, Insights Lead at Square

“A big challenge is when clients are in such a hurry to get ‘answers’ that they’ll compress timelines in a way that doesn't allow for thoughtful analysis. When you try to explain the value of even an extra day, you run the risk of coming across as slow or unaccommodating. It’s frustrating, especially when clients are willing to spend all this money to do the research, but then we often shortchange thinking on the back end because we’re in such a hurry to check the box as completed.”  
— Keith Navratil, Co-founder at needledrop

Strategies for maintaining quality

So, how are researchers giving themselves more time? In some cases they’re doing more work up front, or before a project even begins, to prepare for a project they know will run at breakneck speed. Others pair up with co-workers whose work complements theirs, increasing efficiency.

“I try to shift it to a question of efficiency over speed. Some things just take time to do them well, but there are certainly ways to be efficient about what you do. Tools like dscout and Respondent.io are actually really great tools for driving efficiency rather than speed.”
— Martha Cotton, Group Design Director (Research) at Fjord

“The strategy I’ve used to overcome this in the past is to build a pretty robust toolkit that outlines methods people can utilize to be more agile with their research. For example, having a plug-and-play model for creative screening helps save time on brief writing, methodology scoping, KPI selection and vendor negotiations.”
— Leslie Tse, Brand & Consumer Insights Manager at Google

“We try to build hybrid research approaches that begin with qualitative, followed by quantitative. We’re trying to work more with our analysts to pair up our qualitative findings with the data that they’re seeing from an analytic standpoint.”
— Katherine Lee, Insights Lead at Square

Feeling isolated

Feeling isolated was a problem we heard about from researchers across the board, both those acting as an army of one at a small companies, and individuals at large organizations with massive research teams. It begs the question, how could a researcher feel isolated in a company with hundreds of like minded co-workers? Often, it happened when researchers were embedded on cross-functional product teams. Overall, this structure has had a resoundingly positive impact. With greater proximity to product managers and designers, researchers can build stronger partner relationships. With deeper product knowledge, they can make more informed decisions about when and how research can be most valuable. And product teams can move faster and take collective ownership of insights. But ultimately, the structure does come with a downside, in that it can often leave researchers feeling isolated, without access to mentors or co-workers in close proximity for brainstorming, skill sharing and support.

“One of my biggest challenges is being a solo researcher working across several products and teams. It can be cognitively and logistically challenging to manage the work, deadlines and expectations. While there are other researchers in my extended team, we are all busy and work on different products; I often wish I had another researcher in my immediate product space to run ideas by.”
— Sara Cambridge, User Experience Researcher at Google

“In my previous roles, I have been one of the only quant researchers within my immediate team. The other researchers were either qualitative people, or they utilized mixed methods but lean more heavily on the qual side. The end goal is not different and the synthesis is actually quite the same, but the planning, deployment, and analysis are pretty disparate. And then the goals of the storytelling, and how you visualize ideas or insights is very different, so I ended up feeling more isolated when it came to those activities.”
— Nicole Zeng, Senior Survey Scientist at Slack

“I work from home, so I don’t have my colleagues there to bounce ideas off of or brainstorm different research techniques with. Even at this point in my career, I still come across situations where I’m not completely sure what the best approach may be to get the right insights to my research questions.”
— Kim Flaherty, User Experience Specialist at Nielsen Norman Group

Strategies for finding support

Researchers have developed a number of workarounds to feel less isolated, including forming ad-hoc networks and connecting with outside resources that have the skill sets they’re seeking.

“I found a way to look for brainstorm buddies, advice on analysis, as well as mentorship for further developing my quant skills from different people around the company (on different teams or from different disciplines like a quant-inclined PM or user-focused data scientist).”
— Nicole Zeng, Senior Survey Scientist at Slack

Another tip: If you’re embedded on a product team, think about the areas where you can leverage your team’s help, versus those that require a fellow researcher. The designers and product specialists you sit with may not have a research background, but they can help you carry the cognitive load. Pull them in for tasks like remote moderation, field debriefs and rapid synthesis activities. This not only gives you a buddy for discussion and brainstorming, but it may get them invested earlier in the process.  

Don’t forget: Professional associations like UXPA and The Insights Association are also available as resources. Meetup has groups in most major cities, and the terrific Mixed Methods Slack channel is a newer resource for the UX crowd, full of rousing discussions and recommendations.

Stakeholder engagement... and over-engagement?  

Given how much they have invested in the outcome, it’s not surprising that stakeholders often want to be closely involved in the research process. That becomes a problem when it starts to impact research integrity, whether those involved are internal or external clients. From overbearing stakeholders who try to exert undue influence to get a desired outcome, to conservative stakeholders skeptical of new ideas and methodologies, to clueless stakeholders who don’t understand how to “ask the right question,” stakeholders can create obstacles in the way of research integrity.

“One of the biggest challenges is getting teams to be clear about what they are trying to learn. Also getting stakeholders to look at the broader context of consumer behavior. Teams are often very focused on one situation or behavior, and have a bit of tunnel vision.”
— Bob Burns, Senior Insights Manager at Target

“Given that a lot of people aren’t super familiar with market research and what it can do, they often don’t understand what actually makes a good research brief. There tends to be a lot of ambiguity in terms of what teams expect the research to help answer.”
— Leslie Tse, Brand & Consumer Insights Manager at Google

“It can be hard to overcome the ‘we’ve always done it X way’ mentality when deciding between more traditional research methods and new, potentially better ones. One of the biggest hurdles with transitioning is figuring out the appropriate new benchmarks for success.”
— Ashleigh Penrod, Zeus Jones, Strategist at Zeus Jones

“It’s really frustrating when you’re asked to do a study and a stakeholder insinuates what they would like the results to be. It might be, ‘We’re really excited to hear how people like it’ when they really mean, ‘We really want to see that people like it so we don't have to make any changes.’ That can be difficult and awkward because it puts you in a weird position maintaining your research integrity. I think this is something that every researcher experiences at some point.”
— Gina Knox, User Researcher at Intuit

Strategies for enlightening stakeholders

Many researchers turned to education to help navigate and mitigate stakeholder influence. In some cases, making a few small concessions to stakeholders can also go a long way, and can be done in a way that doesn’t impact the overall findings.

“I try to get to the bottom of why exactly they’re asking for me to make a change to the research plan. I’ll use it as a chance to kind of educate them on good methodology and I’ll try to find a way to squeeze in what they’re asking for, as long as it fits in with the rest of our research. Hopefully then they feel satisfied and it doesn’t impact our research in a negative way.”
— Kim Flaherty, User Experience Specialist at Nielsen Norman Group

“A lot of our clients are not aware of all of the advances in the field and how far research has come over the last few years. So they often ask us to use out-of-date tools or methodologies. For example doing accompanied shopalongs. The good news is now we’re showing them lots of examples of work that we’ve done, including with dscout, that shows an output that’s way beyond their expectations. It can help them see that there’s actually a better way of approaching this issue.”
— Myles Proudfoot, Managing Director at ChaseDesign

Another tip: if your stakeholder is driving the research toward an expected outcome, bring them into the research prior to delivery. By taking them along for the ride, you can gradually expose contradictory viewpoints and help temper surprise or resistance when the insights aren’t want they want to hear. And don’t be the only voice in the room when it comes to presenting your findings. Incorporate user videos, audio quotes and verbatim feedback to build empathy and give context to the users’ situation. Then, once your storytelling has set the stage, offer quantification that validates your insights and checks your stakeholders bias. It’s hard to argue with voice of the customer plus numbers.

Thanks to the researchers who gave us their candid thoughts on this (sometimes difficult) topic. As we continue the discussion on challenges researchers face day to day, and the solutions they’ve developed to navigate the obstacles to successful research, we’d love to hear your take. Tweet at us or catch us on email.