Sample Strategically: How to (Quickly) Find Your Best-Fit Participants

In this tutorial, Ben shows you how to screen user research participants through a strategic lens using a project about fandom.

To get added to the Get Your Geek On project and follow along, head here.

Otherwise, here's a quick rundown of what you'll learn throughout the video:

First, you'll learn about definitional certainty (0:57). To what extent do you need to come right out and ask for the thing you're looking for? Are your users' definitions around what you're researching the same as yours?

With strategic sampling tip #2 (2:56), Ben will walk you through how to insert a small piece of the activity, behavior, or thing you'll want participants to do in the fieldwork in the screener.

Lastly (4:47), we'll challenge you to ask fewer questions during user screenings and show you what this looks like in practice.

Transcript

Hey, it's Ben from dscout. Welcome to another tutorial. Today, we're going to talk through screening and how you can go from instrumental, that is should I qualify or disqualify this person? To a more strategic, what might I learn about this person method of screening? And we're going to be using a project that I'll add you to if you'd like about fandom, a topic that has a lot of different definitions and whose qualification criteria might be more challenging than simply do you use this? Do you not use this? And so the project that you're looking at now is using dscout's recruit tool. It allows for a fully customizable screener, which offers you as a researcher or us as researchers a lot of flexibility. And it's in that flexibility that lies the opportunity to do more with our screening. So let's get started looking at some of the ways we can make our screening more strategic.

One of the first things to consider when screening more strategically is how you're defining the concepts that you're presenting to the participants. And in particular, to what extent do you need to come right out and ask for the thing you're looking for and how might it be more useful or fruitful for you as a researcher or a business to ask what this means to them definitionally? Are their definitions the same is yours? And so here in this example on fandom, I could have started by saying, how many times in a week do you visit a concert for a music group? Now there's not only a lot of potential lead-ins there. A participant might say, "Hmm, I want to be qualified. So let me select that." But I'm also not being as clear or inclusive as I could. I might be leaving out folks for whom concert doesn't resonate in the same way that I want it to.

So definition reality is really important. So in this first question, I've asked them simply, what are you a fan of? No, I'm not defining fan. I'm letting that be open because that's important that I capture lots of different interpretations of fandom. And you can see these red Xs. I'm terminate on folks who are not fans of, or mark particular kinds of categories. And then they move on to walk through, or rather select which of the following things they've done recently. And in particular, I'm very interested in folks who have attended live events and for whom now in pandemic times, aren't able to do so. So again, instead of simply starting out and saying, how often do you attend concerts for music groups? I'm whittling it down while not tipping my hand and leaving space for different sorts of experiences to be involved. So something just as simple as that, adding one or two questions that doesn't presuppose that someone understands what you mean goes a long way toward being more strategic, as opposed to just screening.

Another way to think about strategically screening is to consider the activities, behaviors, or the thing that you're going to want your participants to do in the field work and how you might mimic a small piece of that in the screener. For example, if you want people to take you along with them, as they commute each day, you might ask them in the screener, take a video on your commute or talk about a time when your commute was helped by, supported by a thing or hindered by a thing. Try to test them, or have them try out for your field work, using a media prompt, like a photo or selfie style video in your screener.

Let me show you what I mean here. Here are the participants that I've got as a good fit. And many times they were a good fit, because again, I ask them in a video to talk through, describe, contextualize their fandom. And I did it with a selfie style video because I am going to ask in the field work for them to show me moments and to describe how that fandom is enacted, especially when they're not able to safely go to and engage with the fandom at concerts and events and such. So I'm using a simple selfie video to not only assess their fit, their creativity, their alignment to the various needs that I have for the study, but it's a serving as a great test ground. How are they going to do with the activities that I want them to do in the field work? So you get data that can give you a gut check sense on what's happening with fandom. And it's also a really nice preview to how they might be as participant in your study.

So think about what you want people to do in the field work. And can you mirror that? Can you get a taste for that, a sample of that in the screener? You'll get a whole heck of a lot of information about their fit and the stories that you might be able to tell with the people that you recruit for your projects.

The last consideration, when moving from instrumental to a more strategic, more holistic approach to screening is really something that some researchers have a challenge with. And that's asking fewer questions. I know that research, especially qualitative research can feel very precious. You might not have the resources to conduct new data collections very often. And when you have a sample of folks, you really want to get as much as you can. I totally get that. It's vital to learn as much about our users as possible to provide them with the best possible experience. And it can also be extractive if we're asking a lot of questions that might not help us make the decision to include or exclude someone from the study. So what does this look like in practice? Well, if you have demographic or sociographic or knockout questions, put those at the front, put those at the very top.

If you know you have to terminate someone because of a particular behavior or a criterion that they need to have, put that up front so that they're not finishing the screener only to get to the end after they've completed a lot questions to learn they're not going to be included. Now, if you don't have very hard terminates on demos, that's great. Focus on behaviors. What activities, what doings do you need to have people show you? As I've walked through here, I'm really interested in people who regularly used to at least attend concerts for music. And that's it. Everything else is really up to me as the researcher to understand how is their definition of fandom? Are they a kind of fan that I'm interested in including in this study? And so in this way, I'm a lot about fandom before I even get them into the field work part.

And I'm not asking them anything additional. [Ackman's 00:06:22] razor is something that's used to describe when a theory is good, right? It's parsimonious. It's tight. It's simple. I think the same sort of criterion should be applied to screening. If you can't come up with a reason, a good reason to ask a question, don't ask it. Try not to be extractive in your screening. Qualify, learn about the person, amplify voices that you want to, and then really take time to dig into the questions that you do have to figure out the kinds of stories that you want to tell, the people who you want to represent the recommendations and the decisions that you and your company are going to go with. So to wrap things up, definitionality, don't assume, mimicking their screener to the field work, and being very, very careful with the kinds and types of questions that you ask can go a long way toward making your screening more strategic. And quite frankly, more useful to you and your company.

If you want to check out these data to poke through the various features that I didn't get to tell you you about like analysis, exports, video reel making, and the like, check out the URL on your screen. Thanks so much for your time. I'm Ben from dscout and I'll see you again soon.