Hire the Right User Researcher: How to Write a Compelling Take-Home Challenge

I've written extensively about whiteboard challenges, but what happens when you get a slightly different take—the take-home challenge? I’ve received questions from both sides of this scenario:

1. How do I write an appropriate take-home challenge for a candidate?
2. How do I complete a take-home challenge?

Fortunately, I have been a hiring manager writing take-home challenges and a potential employee completing them. So I've seen this complicated process manifest for both employers and potential employees.

Many recognize candidates' stress during the job process, but we miss the other side. Interviewing and finding the perfect person for your role is very challenging, especially when it comes to the balance of creating things like take-home tests.

It can be discouraging to receive assignments that completely miss the mark. Or worse, work that has been plagiarized. It takes a considerable amount of time to write and review homework assignments, so it can feel disappointing when it seems that no one "passes" this stage. That creeping doubt of "is it me?" becomes more apparent.

Creating a take-home test is a fantastic way to assess candidates' skills, but this can add more complexity to a process that is already highly stressful. So let's try to strip back some of that complexity!

Before creating a take-home challenge

After writing my share of take-home and whiteboard challenges (good and bad), I definitely learned a few things.

A few things to think about when creating a take-home challenge:

Understand the why

The first thing to ask yourself is, “Why am I making a take-home challenge?”

What are you trying to get out of the candidate/what are you trying to understand about the candidate? I see a lot of companies creating take-home challenges just because others are doing it.

Design challenges have been around longer than user research challenges, so they aren't as widely practiced or spoken about. Don't create one just for the sake of doing so because the quality of the test will suffer and likely confuse the candidate.

Some great reasons for a take-home test are:

• To assess how they would approach a problem they are less familiar with
• To understand their process beyond projects they have worked on in the past
• To try to simulate how they would work within the confines of your company
• To consider their cultural fit within your organization (ex: collaborating with others)

Make sure you have test criteria

Similar to the above, make sure you have criteria that you can use to assess a candidate's test. For example, what are some of the key concepts you want the candidate to tackle? What are some "red flags?"

Not only will this help you get clarity when writing the test, but it will also ensure consistency over candidates. The worst take-home challenge I ever wrote, I forgot this critical step, and both the candidate and I were confused.

Some criteria I like to set for take-home challenges are:

1. Does it seem the candidate did any background research on the organization?
2. Does the candidate explain their entire process? Is it a thoughtful and valid approach?
3. Does the candidate challenge the request or jump right into a solution? Does the candidate reconsider the wording of the request or goals?
4. Does the candidate consider collaboration? Talking with and involving other teams and stakeholders?
5. How does the candidate deal with request prioritization?
6. Does the candidate talk about sample size/segmentation and the reasoning behind recruitment criteria? Does the candidate consider how they will recruit the participants?
7. Do the suggested methods make sense for the research question?
8. Does the candidate consider a mixed-methods approach?
9. How does the proposed project fit into the given timeline?
10. Does the candidate talk through (or skip) analysis and synthesis phases?
11. How does the candidate respond to questions and feedback from us?
12. How well would the suggested approach work in our company given the current process and situation?
13. What are some areas for improvement for the candidate?
14. What, if any, are some "red flags?" For example, diving right into a solution, not collaborating with stakeholders, not being able to speak through a solid and thoughtful research process, taking too long to present

Make it about your organization

This point may be controversial, but I have found my best take-home challenges are about where I am currently working. If I am trying to understand and assess fit within my organization, I can better do this on a topic relevant to my work.

For example, I would take a user research project from the past and ask how the candidate would approach it. I have something to benchmark the test against with this approach rather than a spontaneous and new project.

Additionally, using an organization's problem makes the interview discussion much more lively and realistic. Instead of responding to the candidates' questions with assumptions or "I don't know, what do you think?" you can have a conversation about the problem space.

This conversation can be much more enjoyable as an interviewer and the candidate, as it can give them a better idea of how your organization works. Using a past project also makes the test easier to create!

Ensure candidates can finish it in a few hours

I have seen take-home challenges that would take considerably longer than the company estimates. It is one of my biggest pet peeves.

Keep in mind that when candidates are applying to jobs, they might have one (or several) other jobs. If you give a candidate an impossible-to-accomplish timeline, you could be missing out on great talent.

So, how do you assign the timeline and number of hours? Generally speaking, I give the candidate a week to complete the take-home assignment. This timeline means that they can use time over the weekend if strapped for time during the week. I always tell candidates to let me know if they struggle to complete the task in time.

Now, on to the next part, how long should the task take? While we can force whiteboard challenges into a timebox, take-home tests are different. Some candidates, despite the time limit, will spend more time.

The best you can do is assign a task and time limit that align. I like to create take-home tests that the candidate can complete in about three hours. This means that I won't be asking the candidate to do any interviewing, which I now see often!

Before I became good at recognizing how long a task should take, I practiced the assigned tasks. For example, could I complete the challenge in three hours if I had none of my insider knowledge?

I asked friends and colleagues to help assess this as well. And at the end of the take-home challenge, I always asked the candidates if they could complete the task in the given timeline.

Consider paying candidates

Paying candidates for their time to complete a take-home challenge is becoming more common. I've seen several companies raise the bar on this subject. Technically, the candidate is doing work for you.

Of course, you don't have to pay the candidate, but it is something to consider with the candidate's experience. If you make the challenge five or more hours to complete, you should heavily consider compensation.

How much should you pay? How much would you pay a research participant for the same amount of time? I use that as a jumping-off point and discuss the amount with my team and the legal team. There is no one correct answer here, so it is up to you to explore the options and consider your budget.

Create a user research take-home challenge

Now that we've considered everything (and if you're still convinced), how do you create one of these challenges? Well, good news, this is the fun part! Here is how I go about creating a user research take-home challenge:

1. Set a time limit. Before I do anything, I put the amount of time I want candidates to spend on the task. This time is usually no more than 2-3 hours.
2. Brainstorm previous problems. As mentioned, I use previous projects I (or colleagues) have tackled at our organization. I like to pick projects that would show the skills I want to assess. For instance, if I am looking to evaluate mixed methods skills, I would choose a project where I had to use mixed methods.
3. Give context. This candidate will have little insider knowledge, so give them enough context to complete the challenge. This information can be about the position, colleagues, or your company.
4. Include tips. Especially for lower levels, including a few suggestions of your expectations (ex: make it into a presentation) will help the candidate understand what you are looking for

An example

I will pretend that I work at a company that sells smart home devices (ex: similar to Alexa, Google Home). I want to hire a mid-level user researcher to join my team. Here is a sample challenge I would send to the candidate. Check out the example and template here.

Intro

You should not spend more than 2 hours completing this assessment.

The object of this assessment is mainly to assess your thought process and user research skills. It is about your approach, not a solution.

Context

You are a user researcher at SmartHome and working with the Voice Assistance team. You also have to support two other teams with their research requests.

This team is responsible for helping those with medical or physical disabilities to utilize their smart home devices best. They have four focuses:

• Detecting if the user is in trouble (ex: heart attack)
• Staying in touch with family, friends, and caregivers
• Helping with daily medical tasks (ex: remembering medication)
• Helping with tasks difficult for the user (ex: grocery ordering)

The majority of customers are in the United States.

Assignment

Imagine the Voice Assistance wants to understand how caregivers interact with the SmartHome devices to keep in touch with and help their clients. So far, the research has only been on the device owner's side rather than the experience caregivers have. The team would like to present the results in six weeks.

Review the request and create a research plan to help the team.

Tips

• Please elaborate on how you would approach the problem and your particular process.
• There is no right answer or perfect solution
• Consider your audience in each step
• Keep in mind that you are also on two different squads
• Be prepared to present your research plan to us (a 30-minute presentation) during your following interview

A word on levels

Different levels call for different types of tasks. For example, someone applying for a senior role will be able to think through much more complex problems than an intern or junior position. So please keep which level you are looking for in mind when creating a challenge.

I usually think about it this way:

1. Intern/Junior: A basic problem that focuses more on supporting other roles in the organization, such as a mid-level or senior researcher. I air more toward evaluative research tasks.
2. Mid-level: A problem that requires autonomy and thinking through the end-to-end research process with little support from others. This problem also includes collaborating or pushing back on the request. I like to put in a mix of generative and evaluative research.
3. Senior: A more complex problem or juggling several requests at once. I often look for mixed methods approaches, pushing back on requests/educating stakeholders, the need for prioritization, business/team goals, and limited resources.
4. Manager/lead: This goes from operational to managerial, so many requests come in while leading a team with limited capacity. Need to look at the bigger picture of company-wide goals and how research supports that and employee health.

Go forth and write awesome take-home challenges!