UX Research That Serves Communities

In a difficult and challenging time for society as we know it, we believe it's crucial to shed light on those working in research roles that help others in need.

So we brought together researchers at local, state, and federal groups & nonprofits to share how their research serves their target communities—and how yours can better serve your audience, too.

Learn how their work impacts lives and livelihoods, how it's shaped by the unique and transformative moment we're living through, and thoughtful strategies to ensure your research makes a difference.

Taranamol, Nate, and Aditi will share:

• How their work is different than traditional, business- and capital-focused research
• How the COVID-19 pandemic and response to police brutality are shaping their work
• Strategies for making UX and design practices at your org community and service minded

Transcript:

Ben:
Let's jump in to who I am pleased to be speaking with today. Now, these are folks who are themselves People Nerds and had expressed interest last year in talking about some of the work that they're doing. And I'm selfish and needy, and so I thought we could get them all together and have their brilliance multiply. So, Taranamol works as a researcher, thinker, and strategist at Code for America. And each of them will talk a bit more about their work and how they believe it to be service and community oriented, to kick off our discussion. Nate works for the city and county of San Francisco. And Aditi works as a senior designer and researcher at 18F.

So we have three folks who, in their backgrounds, interface with communities in different ways at different levels. And here is, hopefully, what we'll be getting to today. We have a lot to discuss, as well as leaving time for your questions. So I was hoping we could begin with outlining what both I mean when I was writing, and thinking about the Civic UX service-minded approach to research. I'm very interested in hearing from these experts, these folks who are living it and doing it. So they'll each, again, talk a bit about what they're doing right now at their present place, and how they see that work as service-minded. There are some very interesting approaches that they have for things big and small, from research operations, how they talk about and think about recruitment, to incentives, to, of course, their design thinking process. And for those of you out there who are right now in the for-profit, private sector, we'll be doing some comparing and contrasting to what public sector designed thinking and UX strategy looks like.

We will of course touch on the current moment. And I use that word opaquely because I want it to be interpreted by our guests. How it is affecting them and their workflows. They'll talk about shifting projects, or ways that their workflows have been modified. And of course, we'll solicit feedback from you all in the audience on that as well. And then I asked them to come up with suggestions or advice, or something that they could impart on you all if you are sitting there, or standing, if you're doing your standing desk. Or I don't know if there's any of you at a rowing desk, that would be super dizzying for me. But if you're out there thinking, "Gosh, I would love to impart and use, leverage, some of the things they're describing in my org, in my system, with my stakeholders." They'll have some things that they can share with you that might get you, certainly not the answer. I think that's something that will be a theme throughout this. There are very few answers, but there are perspectives, framing, frames, and standpoints that they bring that are invaluable.

So with that, I will stop sharing my screen. And we will become floating heads on yours. And so let's now move to have each of our participants do a maybe fuller introduction of themselves, and talk a bit more about what they're working on. So whoever would like to start, we could do alphabetized. Why don't we go with Aditi, why don't you begin with telling us a bit more about your work on 18F, and what you've been thinking about.

Aditi Rao:
Sure. So, yeah. My name is Aditi Rao. And I'm a researcher and designer, senior researcher and designer at 18F, which is an office within the general services administration, which is an agency of the federal government. And we essentially work with other federal agencies, partner with them, to help improve the user experience they offer. And, the specific focus of my work as a designer and researcher is really actually moving more towards the research side, so it's really about conducting contextual, qualitative, human-centered research. And in our case, that's both with the staff at the agencies that we are partnering with, and with the communities that they're serving the American public at large. Because oftentimes, we're working on technologies, or products, or services that the staff are using in order to do their work.

Aditi Rao:
And then, we're using that research to inform strategy and design. One of my focuses is also how to communicate that visually, and help better disseminate the learnings from our research. Which I know is a common theme for researchers wherever we are. And then, the last thing I'll say is I came to 18F also to build up our work in the area of service design. So I coach folks on that, and I'm working on some efforts related to that. And, I also co-lead ... This maybe sounds like a lot. It really isn't that much. But I co-lead ... We have a research guild. And so that's a group of partitioners, or just folks who are interested in research, who aren't in a research role, where we try to advance the human-centered research practice within 18F. And actually within our parent organization, which is called TTS, Technology Transformation Services, that contains many other organizations doing related work.

Ben:
Thank you very much. Nate, you and I know one another from some of your time at Turo. But now you find yourself at the City and County of SF. What sorts of things are you doing there?

Nate Mahoney:
Yes. I'm really excited for this conversation today. This is really personal work for me. I moved to San Francisco in about 2006. And I'm a queer man in San Francisco. And San Francisco has been the most amazing experience for me. I have learned so much, I've grown so much. And I really wanted to figure out a way of how to give back. I've been advancing my career in the private sector as a UX researcher, and then I've also been doing personal work, helping understand how do people acclimate to this city? I'm really inspired by this city specifically, San Francisco. It's so unique. And so when this opportunity came up to combine my professional work as a UX researcher with my love for this city, I mean I was even doing intercept interviews with tourists who had come to the city to try to understand how do people deal with the urban environment? And so this was just an amazing opportunity for me to just combine my personal interests, my professional interests, to work for San Francisco Digital Services. And it's a city team that works on the website SF.gov.

Nate Mahoney:
And what this website is is it's one website to combine all the efforts. We have over 200 websites in the city. So that's a big part of our work, is doing a lot of collaboration with fellow departments in the city to figure out how we can unify our services. So when someone needs a permit, they don't need to go to five different departments. We can actually have one form. So that's a lot of the work that we're doing. And we're also thinking of other ways to put a lot of these offline services, where you had to go in person, or you had to call someone up on the phone. How do we talk all of these analog, old, offline services and make them digital, put them online, and have digital services that help people in this city accomplish their goals? Get a permit? Get the services they need to thrive in the city.

Ben:
That's great. I look forward to digging a bit more into the different ... I don't even want to use the noun yet, folks. Because I think the way that these folks on the panel approach the humans with whom they interface. I know, Aditi, you're looking. I don't want to say recruits or participants. We'll get more to that later. And again, how they approach and inform and frame their work. But, Taranamol, your work at Code for America. Some of my colleagues have been excited and delighted to work with the Code for America organization. Would you talk to us a bit more about the things you're working on?

Taranamol Kaur:
Yeah, definitely. I just want to say thank you so much for having me in this space. My name is Taranamol Kaur. My pronouns are she and her. I'm a design researcher at Code for America on the GetCalFresh team. And, if you're not familiar with Code for America, we are a non-profit. And we work with government to improve the service delivery of programs. And that is mostly within the safety net, and the criminal justice system. But we also have an amazing brigade system that is in 80 ... I don't even know how many cities across America. That is volunteer led, that is community-based, that is doing work in their local communities. As well, in addition to, the San Francisco office.

Taranamol Kaur:
So, my role is really on the qualitative research team, which sits in the service design team along with the design and client success team. And it's a really interesting intersection to be working with government. But not for government. So it gives us a little bit of a leeway to be able to say no to things. To be able to say no to the things that we maybe don't agree with the government on as well. And, it's also really interesting to be in this intersection of design, and service, and tech. And where can tech do something and where can tech not do something as well?

Taranamol Kaur:
People on my team will always hear me say that tech is never the answer. And it's definitely never the answer alone. But it's really, really interesting to me, in a space where it can be a vehicle for something. But it cannot be everything. And the end all of anything. So yeah. Thanks for having me here.

Ben:
Of course. Thank you so much for being here. I'm chewing on how to begin asking this first question. I guess I'll keep it high level. For each of you, what was the, either if you transitioned from the private sector to the public sector, or if you've been working in the public sector primarily. What are some of the differences that you've noticed or feeling, either in tactical level, how you approach the work, or higher level? I'm thinking about the product researchers and designers I have the pleasure of working with. They look at outputs like cost to acquire a user, session time. We look at hours of usage over maybe support. These sorts of more capital minded metrics. I imagine that for each of you, those metrics look differently because they have to, you're trying to support, serve, and uplift.

Ben:
So, could you talk a bit about some of the ways that the work you're doing now differs maybe from private sector work? Pardon me.

Taranamol Kaur:
I can try. So, I have been at a design agency before this as well. So I have a little bit of experience in doing work for a product that is trying to make money. And I think the difference in the work that I'm doing now is that during the research process, I have to constantly ask myself, "What am I doing? What is the purpose of I'm doing?" I'm not creating for the purposes of monetizing a product, or making money. Instead, my work is centered on community. It's centered on the people that I am working with. And often, the people that I'm working with are trying to navigate a really, really difficult bureaucracy in order to obtain something so vital, like food.

Taranamol Kaur:
Food is a human right. It should not be hard to get. But, it is tied in with so much bureaucracy, it is tied in with having to prove that you make X amount of money, so you can get X amount of benefit amount within a month for food. And it should not be like that. It's a giant maze, and I think that the thing that really differentiates from the work that I have done in the past, and the work that I'm doing now is that everything that I do, I really question why am I doing it. Am I doing this for the folks that are trying to get benefits? Or am I doing this to answer a research question that may not really need to be answered? So it's really about centering community in a way that I think is so powerful.

Ben:
Aditi and Nate, you both transitioned from the private sector to the public. Were there things that you noticed right away when you got in, and working with your teammates and stakeholders? What stood out to you when you moved?

Aditi Rao:
Sure, I can go ahead and answer this question. I've actually bounced between public and private over the course of my career. I started out in tech long ago. I've been in this field for many years, more than a [inaudible 00:13:02]. Then I actually transitioned into working as an urban designer for several years. Which is essentially working with local city governments, public sector. And then back to the private sector, and now back to the public sector. And really, what I see, and that really excites me about the work that we do within the federal government ... One thing I'll say that's really interesting about our little group here at 18F, within the GSA, is we are very empowered as employees to say, "This is a project I would like to work on, and this is a project I would not like to work on. It does not align with my values."

Aditi Rao:
So I definitely am able to practice my values and really focus on the reason I came [inaudible 00:14:00] government again, or in government public services again, which was to serve people. To serve as many people as possible. To improve the lives of as many people as possible, which was the goal of my work as an urban designer. And is the goal of my work right now. But then looking sort of outside of me, and my motivations or goals, what's really different, and especially when you were talking about all these metrics that folks had been in private sector track, or have to follow, or sort of not just track, but that they have to determine. And keep certain numbers on.

Ben:
Or let themselves know, "Why should I exist?"

Aditi Rao:
Right, right. Let's be honest, ultimately in the private sector, money is the driving factor. A business needs to make money to exist. And also businesses want to be market leaders, all that stuff. So ultimately, the bottom line ends up being money, even when you, as a user researcher, are there to say, "Well, I care about people. I care about the people we're serving with this product." And then I just remember spending a lot of my time trying to show how serving the people would actually improve the bottom line, that was the big thing for me.

Aditi Rao:
But here, in the federal government, I have been ... I think a lot of people can be very cynical about government. I certainly had heard the perspectives. But almost every federal employee that I have worked with cares deeply about the mission of the agency that they work at. I mean, and it is so just inspiring. Every agency has a mission that is about servicing the public. And, for example, right now I'm working on a project about the rule making and public commenting process. And we have completed a number of interviews with folks who do the work of writing the rules, and then taking the public comments into consideration. And, they care deeply about really making sure that they're taking into consideration the public's opinion. And they want to in fact make sure that they're not missing out on important opinions.

Aditi Rao:
People are mission-driven. I guess I am rambling a little bit here. But people are very mission-driven. And that's inspiring. I worked on a number of projects with NIH last year, and we were serving the science community, to improve the lives ... Better science to improve the lives of people. I mean around the world, the work that NIH does and funds improves the lives of people around the world. So that's just been really exciting and one difference.

Aditi Rao:
One other thing I did want to mention about the difference that I think is key in our work is when it comes especially to the services that are the public services for everybody. They are for everybody. We need to serve everybody. They need to be usable, and useful, and accessible to everybody. That means the notion of an edge case in technology is like, "Oh, we can leave out the edge case." Especially in the private sector, folk can say, "Well, let's just focus on the 80% of the market or something." We don't have that choice. And I have this conversation with other colleagues at my office that were real excited that we don't have that choice, were like, "Yes. We have to serve everybody."

Aditi Rao:
And so I just really think ... And it's not just that we have to include them. It's that oftentimes, the services that we're working on, the edge cases are the people who need them the most. They're actually the people that we need to be focusing on, and centering our work on. And, I could talk about this forever and ever. But I think when we include the edge cases ... I should actually probably stop using that term, and come up with a better term than that. That we actually end up serving everybody better. I mean, this has been, I feel like, an accessibility. Technology accessibility that's become more and more recognized. Like, "Oh yeah, if you increase the font size for people who have vision impairments, then yes, it will be easier to read for everybody."

Aditi Rao:
But this is true, I think, beyond just usability. But when you think about an entire service, digital and non-digital, how is it truly accessible and useful, and understandable, to as broad as an audience as you should be serving?

Ben:
And Nate, that resonates a lot with the work that you're brought in to be thinking about [crosstalk 00:19:02].

Nate Mahoney:
Yes, definitely. I mean, plus one to all of this. And then, working for the city is so interesting because there are so many different departments who are our stakeholders, that we collaborate with. And if we're not careful, we can think of them as our client. As if we do what they need as the end goal. And so what's really exciting is in my role, where I really want to be able to give back, and I really want to ultimately help people in the city. It's through research we can frame that back to people. And the people that are going to be using these services and these tools. And so then we become collaborating for the people in the city, instead of just digital services serving a department. So that's where it's been really rewarding, as a researcher, is to help frame that conversation around people and the people that need to get things done. And how can we make that work and bring these services online? And just make them more efficient.

Ben:
I'm wondering if this approach and framing to the work being done has an impact on the methods or tools that you might select. Aditi, you mentioned interviews. And Nate, you had talked about doing triangulation with a survey and pairing it with focus groups. Do you find yourselves being more free to choose methods, or tools, or approaches in the sorts of places that you are? And if you do have more freedom, what are you choosing? And why?

Nate Mahoney:
I mean, mixed methods approach still works really well here. We have a big population, and then we also want to understand why. So, mixed method approach worked in the private sector, it worked here as well. And then the great thing is is we have a lot of partners in different departments that can help us roll out some of these bigger endeavors, especially when we're trying to have things translated. San Francisco works in four different languages, so it almost feels like we are having to localize within the city, we have to translate into Chinese, Tagalog, and also Spanish. So all of the collaboration with our teams, we have a lot of allies. And it's finding those different allies in different parts of the city, where we can feel a little bit siloed. But once we kind of start talking to people and break that down, we can see that there's a lot of similar work going on. And we can actually just combine our efforts to better serve people in the city.

Ben:
Aditi, Taranamol, do you find yourself going toward or reaching for a certain kind of method that matches with the mission for each of you?

Aditi Rao:
You know, I think the truth is, in our work, we're a distributed team. Not just across the country, and we have four offices, but also we have many ... I have many colleagues who work in places in the country where we don't even have an office. So truly remote and distributed team. And so this, for me, was one of the challenges I saw when I came in. Because I really believe in the value of in-person ethnographic research. Understanding people as they are situated in their lives. And in their context.

Aditi Rao:
And much of the research that we do, and because we are remote and distributed, is over the internet. And so, I have made the effort in my work, and then worked on encouraging this through the research guild, of how can we do in-person research in the places that we are? Or, there have been projects where people have gotten budget to travel. It depends on the project. But, for example, when I was working on projects with the NIH, did work to recruit scientists who were in labs working. So that we could visit them on site, and see what is the work like? And then sometimes I think survey is a dirty word in our field. But actually, especially as related to this current moment, and on our project, we actually ended up doing a survey. I do dislike being sort of completely anti-anything. I think there's always a time and a place, depending on the circumstances. If the survey is the best way to reach the broadest audience possible, then I think a survey can make sense.

Aditi Rao:
Yeah, I think we unfortunately do default to this doing video interviews, and I think we are, especially in this moment, recognizing and having more of a conversation around what are other ways in which we can involve people? Can it just be by phone and not by video? And can we still ... Yes, actually. We probably still can learn a ton. And more people probably have phone access than they have internet access. Although I will say, we are using technologies that will do great and allow us to make a phone call if they don't have video available.

Aditi Rao:
I know that, from our previous conversation, I know that Taranamol and Nate, and their work, have been able to partner with community organizations more closely. Some of our longterm projects, they're our teams that have been able to do that. Actually we do end up having some state work, like with the State of Vermont right now. And so it is possible for that project, they are partnering with global organizations. I think it's tricker, to be perfectly honest, if the federal government's like, "We're serving people from across the country. Which local organizations do you partner with?" [crosstalk 00:25:20]

Ben:
Yeah, Taranamol, you've talked about accessibility. Oh, pardon me, Aditi, I'm sorry.

Aditi Rao:
No, no, that's it.

Ben:
Yeah. Taranamol, you've talked about how you think about access to ... Ruben's got a great question at the top of the list here, about the digital divide. And I don't think ... We will address that certainly. But how does access to, or availability of certain services, technology. Because, again, each of you is in the tech space. But tech is still a very narrow slice of humans. Humans you're trying to work with and serve. So Taranamol, how does that factor in and play into the work you and your team are doing?

Taranamol Kaur:
Yeah, definitely. I think the past year or so that I've been at Code for America, we've really, really been grappling with how we do research. And when is interviewing the right time? When is diary studying the right time? And Nate, to your point, mixed methods is great. You have to have a balance of both of that work. We've really been thinking about how do we co-create with people? How do we bring people in? How do they do a whole workshop with us? How do we bring people in, and then we tear apart our whole application? How would we redesign this? How would we really make this ... What are the words that you would use? What are the ways that you would want to see this information? In a way that you want to answer it, and you see yourselves in it, answering it?

Taranamol Kaur:
And I think a problem, for us at least, we've had these conversations a lot on our qualitative research team, is it's hard to just do one interview with a person. Like, "Hey, I'm going to pay you," how much ever we're going to compensate somebody. "We're going to extract all of your knowledge," and then that's it. That's the end of our relationship there. And we've really been grappling with is that really co-creating? That isn't co-creating. That isn't really bringing in community, that's like, "Hey, let me extract all of this knowledge from you." And I think especially in this sector, and especially in this work. In any work that is not okay, but especially in this work, for folks who we depend on to relive their traumas, so that we can understand, and we can make a change in the application. That becomes so problematic.

Taranamol Kaur:
And we're really grappling with, especially within the pandemic. Our research and our client success team changed the way that we do research. We saw thousands and thousands of messages that were coming in from folks who are going through our digital application. And who are telling us, "This question doesn't make sense. Why are you asking me about income? Why are you asking me about jobs? The pay stub that I have right now that you need me to upload isn't going to match what I'm going to get in two weeks." Right? So, us in that moment, being like, "Hey, do you want to get on an interview with us? Do you want to talk to us?" Isn't the right thing.

Taranamol Kaur:
And I think that's what we really have to think about is how do we do non-extractive research? How do we do research that we're not constantly asking people to extract information so that we can make a change and relive through trauma. That is really hard and that is very problematic. So I don't have a complete answer for that. But it is definitely something that we're thinking about how do we change the ways that we do research?

Ben:
Yeah. And for the audience, this is a very thoughtful group. And they are a group of folks who don't want to be too prescriptive, and are very standpoint-driven and perspective-led. And I'm sitting here, again, as someone who thinks, and designs, and does a bit of research not in this world. Each of you had mentioned, and as Taranamol just said, that the extraction-based work of research. Each of you also talked about the commitment to looking at the humans and the folks that you serve as non disposable. Nate in particular, you had mentioned how the extent to which you were able to serve these community members means that they will come back and ask more of you. And I think about some of the customers with whom we work, but by no fault of their own, I think it might just be hidden or blind to them. They are interested in a subset of the population so that they can drive engagement and get users. But they might not be interested in touching back in with those.

Ben:
Each of you, I'm pointing because you're all here. Each of you talked about how it's imperative that you both respect the time of the folks you're bringing in. And then touch back with them. So Nate, I'm wondering if you could talk a bit more about that notion of disposability, or non-replace ability of the folks that you're researching with?

Nate Mahoney:
Yes. Working for the city has been eye-opening because people have relationships with this city that have been ongoing. And have been going for a very long time. For example, I was recently doing interviews with small business owners. And these are third and fourth generation business owners, where they've been working with the city, and will continue to work with the city. So this is where, as a very sensitive person, which is why I'm drawn to this kind of work, I could start to hear these stories about the parents who started the business. And the decisions they were making. And now the next generation that needs to figure out how to keep this business running during a pandemic.

Nate Mahoney:
And this was really touching to realize that previously I had been doing some usability work, and will we need people to get eyes on our product? And this is really a different approach. Where it's like we want to start building relationships with these people so that we can come full circle, and even go back and show them what were we able to do with the information that you gave us? How can we support you? And one of the findings that came up for one of these business owners I was talking to is that the communication that they do and don't receive from the city is so important. When they're hearing from the city, and they're understanding what they need to do, it's a great thing. But, if they're not hearing, that's a miss.

Nate Mahoney:
And so, it was really on me to figure out and followup. One business owner had not heard back about the status of their loan application. And I was like, "I'm not going to drop that." I figured out, I've got to help them find out what is the status of their loan application? And really wanted to follow through on that, so that even if something was broken in the system where they weren't able to have that communication that they need, that's kind of the difference that I'm seeing here. That's really exciting for me because I ultimately want to help these people. And I want to develop these relationships ongoing. Because these people have been a part of this city, and will continue to be a part of this city. And that's what makes it so rewarding as a researcher, is just to know that this is really work with impact. It's work with purpose. And the other people who take on these jobs, my colleagues, they're here for the same reason. We all want to figure it out.

Nate Mahoney:
It's not always perfect, but we keep trying to figure out how can we improve? What can we do better? And ultimately, it is about these sort of more longer term engagements. That's another thing too that's been really interesting is this idea of engagement. Government engages with the community, and then as a researcher, I'm like, "Oh, what is this?" And there's actually this really awesome overlap with engagement and research. And I think that's a real sweet spot to continue to investigate, where a research method might meld with an engagement method. And it really ends up being some language nuance. Because this engagement has been going on for a while, and they're doing really important work. And where they overlap, and where we can work together is very exciting as a researcher for the city.

Aditi Rao:
Yeah, and Ben, if I can piggy-back on what everyone's been talking about. Yeah, I think I was speaking earlier to the neat way in which some ways the research that we do, in our organization, can be a little constructed. But then we also share in our ethos absolutely this ... I mean, Taranamol, you used the words exactly. We talk about not seeing our participants ... The relationship that we have with the people that we're speaking with, as extractive. And that it's not a one time thing.

Aditi Rao:
And in our case, this was especially true because up until literally right about now, we have not been able to pay research participants. And it's because of technical details, legal sort of interpretation of an 1800s federal act, called The Antideficiency Act. Which I actually had to look into like, "What was it? Why was it?" And it's really for good reasons. It's about not using money that the government hasn't actually allocated specifically towards a purpose, and giving it to the private sector. So, trying to stop federal agencies from being corrupt. But unfortunately, legal interpretations were preventing us, then, from paying people for their time.

Aditi Rao:
Finally, there's been some shifts and reinterpretations. And actually a new focus on customer experience, which I'll take the terms as they come. If that means that people throughout the federal government are realizing that talking to people and compensating them properly is important, then great. But, outside of being able to compensate them ... Well both because we couldn't, and because we were having this larger conversation about ethics, research ethics. What does it mean to involve these people who have many demands on their time? And now, in this current time, have even more demands on their time.

Aditi Rao:
How can we, right, not just make this extractive relationship? So how can we continue keeping them involved, how can we understand what it is that they ... Why are they participating in our research? What are they hoping to get out of it? And so that we can maybe give that to them. Involving them on a controlled basis, not just one time. Updating them on where the project is going. I mean the project overall, but also then they get to see how their involvement has impacted the project. And especially if it's a service or product that they are using, or may use, or may benefit from. They can see how their involvement hopefully has improved it. That's what we hope for. And it's not in a way to sell them on continuing to participate in our research. But more to be like, "Yes, this is ..." The truth is, people get a lot from seeing that their involvement has made a difference.

Aditi Rao:
And when we just talk to them one time, and just maybe send a thank you email, and then they never hear from us, they miss out on that. And so now that we can pay people, we're definitely going to pay people. But I think we're definitely going to continue all of these other practices. And I'm actually co-creating ... I mean, we have this difficulty of not being able to, generally speaking, do this kind of co-creation in person, unless it's with the federal employees that we're working with. But, we do do virtual co-creation sessions on various projects to figure out how can we involve our research participants in various ways in our research?

Ben:
This is aligning with a lot of the questions that I hope the panelists can see. There are a lot of really great questions about how can we move to more power or influence offering, i.e. making the folks with whom we're doing research feel as though they're not only involved in, as Aditi is talking about, but empowered by the work. Taranamol or Nate, are there things that you are doing to, as Aditi said, keeping folks involved? I remember back in my graduate research days, I would try to hold debriefs with folks just to let them know where the research was going. It didn't directly impact them in the waves. But they loved, Aditi, to your point, they loved to know what the questions were going to inform. And why I was asking this line of things, or having them do this certain activity. They loved being kept informed.

Ben:
I'm wondering what else you all are doing to empower and engage with the folks you're working with.

Taranamol Kaur:
I can speak to that a little bit. A lot of it is what you just said, Ben, is keeping folks informed throughout the process. And also inviting them back in. Making it not just a one-time thing. And we do a lot of work with community-based organizations. That's where a lot of our recruitment comes from, is folks that are on the ground, doing this work every single day. So keeping them informed, and keeping them a part of the process also helps how we don't just think about participants as a one-time interview thing. But as you're part of this process, this is yours.

Taranamol Kaur:
And I think that also really just goes down to how we see ourselves. We are not experts by any mean, in any domain that we are working with. Right? We're just facilitators in that process. So really letting folks lead the way on whatever is happening is a way that we can make sure that we are shifting that power as well.

Nate Mahoney:
Yeah, plus one to collaborating with community benefit organizations. We've had a lot of success with that. Some of these relationships have been set up previous to my joining. And just seeing that over time, it strengthens our ability to explain what we're doing. Often we're talking to people who have never been a part of user research before. And they don't understand how websites are built, and they just assume it's kind of a one way street. That the website they're using is the one that they have. And so, there's a little bit of a nice moment there, when we're able to make that connection, that we can make changes based on their feedback. And so these relationships are ongoing, and we're building community through these community benefit organizations. So that's something that if you're looking at this kind of work in the public sector, that that seems, at least working with the city, as a really key component in terms of who we're engaging with to get feedback on our work.

Ben:
Thank you to each of you. And I know that that's a broad questions. But I think that that's really useful for some of the attendees who are, again, curious about how you're both conducting the work, getting the work done, to build and improve these various services, products, and engagements. But then again, actively involving. We had a great question from Bernadine that as a former statistician, I guess one never really leaves the statistician title. But as someone who was a practicing statistician, the gold standard was my being able to standardize the outcomes that I had so that any other researcher could read and go, "Ah ha! I know how to take those data and do something with it."

Ben:
I'm wondering if any of you have, again, advice or suggestions, or stories from the work that you're doing on how you're sharing the outcomes to the stakeholders. Internally. The folks who might control budgets, or might help frame up what the work you're doing. Bernadine has a really good question about unifying and communicating. They ask, "You have mentioned improvements, values, purpose, community. What would you say are the ways of standardizing these various practices?" And again, a very big question. But if there's something that relates to you on how you're maybe talking to others internally about the work you're doing so that they can take it and go, "Ah. You're making this sort of progress, and here's how we know it."

Nate Mahoney:
One thing we're doing is just standardizing across teams within digital services. So we have several different departments. We have a housing team, a SF.gov team, a permitting team. And so they've been working fairly independently, but now we're using some tools, like a dashboard, to show research projects across all teams. Because we're finding that there's a lot of similarities. And there can be some of these mega findings that apply to the same form that someone's using that we learned from ... Permitting also works for SF.gov. And another thing that we've done too is we've started a rolling research program. Which involves all of the teams, and they can put in their usability. Usually the valuative kind of questions into a monthly rolling research.

Nate Mahoney:
And that's been great, again, for just sharing out learnings that are ... Break down silos. So we, again, learn that a lot of our work has to do with forms and improving forms, and those are on housing sites. They're on SF.gov sites. They're on permitting sites. They're quite across. And so it's been helping us to think about a design system, and standardization. And also the content for what's in these forms, and how they can work for everybody. And then doing more regular share outs, as a way to share the research.

Nate Mahoney:
We have the beginnings of a research repository in a Slack channel. Where whenever a team finishes work, they do a little writeup, and they post it to the Slack channel. And then you can see this kind of rolling list, and you can go back and see what previous work has been done. So that will continue to grow, and it's a nice way to share. And kind of start thinking about what are some of the best practices that we don't need to recreate the wheel. And what's previous research done before so we don't need to do that again? But maybe just look at those findings.

Taranamol Kaur:
Yeah. And similarly to that, I think there's things that you can standardize, and there's things that you can't standardize. So for how do we keep our research, and how do we share our research, all of that is ... My colleagues are going to laugh, I know they're laughing right now. But Airtable has been amazing for us to really keep a good repository of the way that we are coding, the way that we are sharing insights, the way that we're tracking research to design. Code for America loves Airtable, I can say that.

Taranamol Kaur:
I think synthesis and sharing synthesis is different depending on each project. So a colleague, Cesar, and I, last year, put up a multimedia synthesis where we had done usability testing, and we made it a museum walkthrough. We put a playlist up, which each insight correlated to each song. Cesar did illustrations. So, we learned so much. And sometimes it is hard for folks that haven't done the research to really digest everything. And sometimes a deck isn't the best way. Sometimes a deck is the best way. So it's really about how are we sharing the things that the team needs to move forward? Is it a multimedia museum walkthrough? Is it a deck? Is it an email that says, "Here's a link to all of our research, but here's the top three things if you don't have time to read it that you need to take away?"

Taranamol Kaur:
So, it's really about every time we do synthesis, it's always different. the way that we share out is different. The way it's documented is the same. So if anybody needs to come back and look at our research, that is the same. But the way that we share out is really different every time.

Aditi Rao:
Yeah, I'd like to sort of piggyback on that. I think what I was hearing a little bit in that question too was about statistics, and quantitative research versus qualitative research. And, this has been a personal bugaboo of mine for years, because I actually started out as a math major. I love math, but I also have a big problem with the misuse of numbers. And I have a big belief in the value of qualitative research to tell us things that we simply can't learn through numbers.

Aditi Rao:
And, in one of my previous jobs, I was working in financial services, which just sort of worships the number. And so that was just an extreme constant struggle. But even in, I think, many other sectors, there's just this belief in numbers in a way that there isn't a belief in qualitative research. These questions of reproducibility and all that kind of thing. And I think if you get into the ... I'm going to use a big word. Sorry. Epistemology of how do we know things in qualitative research? It's totally different from how we say we know things in quantitative research. So I don't think you can even compare.

Aditi Rao:
And I try not even to get into that argument, to be honest. And what I, and I think in our work, that we try to do is ... It is pretty nonstandard. I mean it's just team by team, they determine what is the best way to share out their research. And, I'm a really big believer in visual methods and storytelling. Because what we're learning from the people are the stories of their lives, right? And the context within which they live. Within which they then need or they'll use our services or products.

Aditi Rao:
And actually, I also believe in whoever your audience is, whatever way they're used to being communicated to, just go with it. If it's PowerPoint, I'm sorry, just give in. Do a PowerPoint. You know? And so decks are a big thing, so on our current project we're doing a deck, and I'm a big believer in journey maps, but we're actually translating journey maps into like storyboards. We're telling this story that we might tell on our journey map through sort of a storyboard narrative. To really humanize what we've learned. And actually what we do is incorporate, and I know a lot of people ... I've seen other projects or examples that do this. We incorporate statistics that we've learned, that back up what we learned qualitatively. So people can see that there's a correlation between these numbers that they know. But then they see them situated in that context.

Aditi Rao:
And I think that's something that actually we are talking about more at 18F. Actually mixed methods. Nate mentioned it earlier, mixing qualitative and quantitative. How can they support each other? How can they play off of each other and not sort of be at odds? And, yeah. I guess that's probably going to be the answer.

Aditi Rao:
I've been wanting to say something. Ben, this is maybe off topic, or I don't know. Maybe you asked a question and I missed it. But I've just been wanting to say in this moment, we were talking a lot when we were preparing for this, about how are we approaching research in this moment? And I just want to say one thing about that is that we are asking the question, "Do we need to even be doing research in this moment?" At this time, when people are so ... There's already bias in our recruitment. We know that. The people who are able to participate are the ones who have the time and the energy, and resources to participate, right?

Ben:
Sure. Or the accept, [crosstalk 00:49:51].

Aditi Rao:
And that's just only heightened at this time. And we are just even more asking the question, "Do we need to be talking to ..." And actually, Taranamol did sort of, I think, touch on this. "Can we be learning things in other ways? Can we do our research later? Can we do hero stick methods to evaluate what we're working on? Can we do surveys? Do we research on the internet where the types of people we're interested in talking to congregate?" And learn by reading what they've already written elsewhere. Just being really creative about that, and I just wanted to share that. That may be totally off topi.

Ben:
I'm so glad you did because we have about five minutes left, and my last question in my notes was if each of you could impart something. And so I think that's such a great way to tidy up if someone is attending. And very briefly. To the attendees who are in the chat, I cannot thank you enough for that engagement. As someone who bothers you all a ton, not only our guests, who have given so much of their time, effort, and brilliance. I cannot thank you all enough. I know I'm asking an hour of your time to sit in, and I hope it's useful and interesting, engaging. But to you all who are in that chat, we're going to be following up People Nerds, publishing all these things. Questions about Eric Table and various articles. But your work will not go unnoticed or unrecognized. Thank you so very much, selfishly. You made my job so much fun.

Ben:
So if I could in the last moments that we have turn to Nate and Taranamol, if there's something that you could say, like Aditi said. Wherein, do you need to do research? Is there something that you might impart to someone who's either thinking about the current moment, and whether or not ... Just thinking about this current moment? Or something that they can be doing to make their team, org, or business more civic-minded, community-driven, service-led? Taranamol?

Taranamol Kaur:
Yeah. I have a few pieces of advice. I think the biggest for me is letting go of ego. Which has really allowed me to be in community. This is something that has been instilled in me since I was little, it's one of the things that my faith talks about. That it's not about us, but it's about us as a collective. And, to do good, to do real good work that is truly in service, one has to remove themselves from the process. One has to center around community. In what my faith is called Sungut, and this concept of Sungut and service, of community, is to whoever we're building, creating, and being in space with. That has to be at the center.

Taranamol Kaur:
And while I've been at Code for America, we've been learning this. And our qualitative research team, our service design team, has been grappling with these concepts of what does it mean to do good design? What does it mean to do good research that is in service? And not in the ways of human-centered design, but where we're trying to create a solution, or find an insight, or get x amount of clicks on this button. It's not about that. To be in real service isn't about those clicks. Those measurements, I think Nate and Aditi mentioned this, those metrics and measurements don't exist in the same way in this domain. And it's about how do we sit and talk, and be with people? How do we really listen to them?

Taranamol Kaur:
And letting go of ego, I know that has nothing to technically do with design. But letting go of ego has to be the center of that. And it can't be about me. I don't have to be right all the time. I don't need to have all the answers. I don't need to be productive. Aditi said that. That's where capitalism comes in. We're brought up in a society that what we produce is what our worth is. And in order to do good service design, and research that isn't service, it's about us as a collective, and how do we grow together? We're not the experts, we don't need to be the experts, but it's about us, how we are in community.

Taranamol Kaur:
And my other piece of advice is to read. Read books by BIPOC communities. Read by women of color. Read by people who aren't privileged to be in the normal design sector. I know a lot of people mentioned Creative Reaction Lab. Please, please read. If you're interested in this, read about Creative Reaction Lab. Read about emergent strategy. Read the colonizing methodologies. Read all about love by Bell Hooks. This concept of UX and human-centered design, it's not new. It's existed long, long before it was coined as UX or user experience, whatever you want to call it. And building around community, and love, and healing as a collective has existed long before any of us have been doing this work. So learn from those people that have been doing that work. Those are my few pieces. [inaudible 00:54:46].

Nate Mahoney:
I just want to jump in and say that work for the city of San Francisco right now, in digital services, means that we've had more work than we've ever had before. I mean, it started out just getting information out on the website so that people can be informed. And then it switched over to putting digital services online, because everything's closed, and people need to have services online. And then the big piece of thinking about a civic-minded approach is who are you engaging with? And how do you engage with community benefit organizations? That's been the real big thing for us. Is to just be really doubling down on that. And really thinking very carefully about who we're talking to, when we're talking to them, how we're reaching out to them. And building these relationships. And I think that's the thing that's, if you're looking at this civic-minded space, that's one of the key areas that has been different for my experience of working in industry before.

Aditi Rao:
And Ben, I know you thought I gave my last answer, but I can't. That actually ... I did have another answer.

Ben:
[crosstalk 00:55:54] all day, please Aditi, please.

Aditi Rao:
I'll be really quick.

Ben:
No, no, no, no, please.

Aditi Rao:
The edge case is actually what you should focus on. That's one thing I want to impart. That will improve your design so much more than focusing on the majority, quote. And the other thing is, especially in this moment, I want to recognize and really say explicitly, we are going to be less productive than we normally are. There is a huge pressure already on us as researchers to produce, to produce, produce insights. And, to prove our worth to people who don't have an understanding of our work. And I think, especially in such difficult times, all of us as researchers, all of our colleagues are being impacted by these ... I'll just say difficult times. As well as our participants.

Aditi Rao:
One of the things we're doing is taking longer to sort of talk with folks in the beginning. Give them time to sort of, if we are doing interviews or research, to give them more space to share whatever they feel they need to share. So we understand what they're dealing with. That means we're going to get less useful, quote, insights. That's okay. And that's what I want to say. Being less productive in this time is okay.

Ben:
I'm so glad you did. I'm so glad you did. And, again, my eternal thanks to Aditi, Taranamol, and Nate for just some of their time. For those of you who attended, find these folks on LinkedIn. They are brilliant, they are smart. There are so many resources in the chat that we'll be sharing out. So thank you all for joining. For my panelists, it has been an absolute pleasure. Humbling and educational in all the right ways. For me, as both a human in this space, and as someone working in this space, thank you for teaching me quite a bit. And I was delighted that we got to share just a bit of it this time

Ben:
So we'll hope to see you all in the next People Nerds webinar. And thank you again. [crosstalk 00:58:04].

Aditi Rao:
I just wanted to say thank you, Ben, as well, for organizing this. You're really wonderful.

Ben:
Oh, of course. I'm so glad that it wasn't a total drain. I know that, just as we ask our attendees to spend some time, [crosstalk 00:58:14].

Nate Mahoney:
Thank you so much. Thank you. This has been a pleasure.

Aditi Rao:
A real pleasure.

Ben:
I hope that we can not socially distance meet soon. My colleagues will tell you, I'm a big hugger. So I am needing some human contact. So I hope we get to meet pretty soon. Thank you for-

Aditi Rao:
And I'm saving the chat because I didn't have the time to read it, and I didn't-

Ben:
I'll share it out with everyone.

Aditi Rao:
Oh, cool.

Ben:
I'll make sure. I'll share it out with everybody, yep. Okay. Keep well y'all.Take care.

Nate Mahoney:
Thank you.