How to Conduct An Effective Card Sort: A Comprehensive Guide

Card sorting is excellent for understanding mental models and how people categorize information. It also serves as a way to visualize the skeleton of a website or app.

However, as with many qualitative user research techniques, many questions arise when creating one. Card sorting can be an extremely subjective process.

I want to highlight some best practices and tips with a concrete example of how to approach a card-sorting problem.

In case you missed it! 

Closed, open, and hybrid card sorting are available on Dscout. Learn more about where you can use these methods and see our platform in action.

What is card sorting?

Card sorting is an activity in which you give “cards” to a participant and have them order it in a way that makes sense to them. These cards can have information written on them, be blank, or be a combination of the two.

Below are three different card sorting techniques you can use in your research.

1. Closed card sorting

Closed-card sorting is when each participant gets a set of cards with information already written on them. They are limited to using these cards.

This approach is very evaluative and is best when the terminology or concepts are well-defined and established. It can give you apparent patterns on the cards.

However, the significant cons for this are that you might not fully understand the user's mental model, as they have to conform to what you wrote on the cards.

2. Open card sorting

Open-card sorting is the opposite of closed-card sorting. Participants create categories and concepts of their own and then order them.

Open-card sorting is great for exploratory work, and understanding how users relate to, organize, and define different concepts. It can lead to a better understanding of terms and definitions.

However, the con for this approach is that the patterns are usually not as straightforward as closed sorting.

3. Mixed (or hybrid) card sorting

Mixed-card sorting includes cards with predetermined information, but allows the participant to create new categories or concepts that may be missing. When I do mixed card sorting, participants can "edit" the pre-written information.

Mixed card sorting is my favorite because it allows for both evaluative and generative work, although you may still run into the same cons of fewer distinct patterns. The mixed card sort generally calls for a more significant number of participants.

How do you choose between card sorting techniques?

First off, determine if you are in a generative or evaluative phase of the project.

Would you feel comfortable and confident in defining terms, concepts, and categories for your users? Have you conducted previous research that would help you correctly identify those terms, concepts, and categories? If yes, you can then use closed-card sorting to evaluate the patterns better.

If you are starting from scratch, or don't feel confident in creating cards, I recommend going with either open or mixed-card sorting to first understand how users define these areas. Then you can follow up with a closed-card sorting after that.

If you know that some concepts are validated, but others not, then go for the hybrid. If you want to go all out discovery, open will suit you best.

Regardless, make sure you always capture the running commentary as the participants are placing, writing, or organizing cards.

Moderated versus unmoderated card sorting

Moderated card sorting involves a researcher being present in the room while the user participates in the card sort. The researcher will encourage the participant to think aloud during the process and probe into why they are making certain choices.

Another way is to interview users post-test to gain extra insight into their decisions and ask questions about specific cards, if necessary.

On the other side, unmoderated card sorting is when the user does the card sort alone. Users typically perform unmoderated card sorting online, with the help of a card sorting tool. Some design teams employ this method since it tends to be a quicker, cheaper option.

Example:

As I mentioned, there are a few use cases for card sorting. The most common goals that align with card sorting as a methodology are:

• Evaluating a users' mental model on the information architecture of a product/service
• Understanding how concepts relate to each other in the mind of the user and the hierarchy of them
• Uncovering definitions, terms, or ideas that might be missing or misunderstood

As an example, say we are working at Google, and the team from Google Shopping wants to re-imagine the Google Shopping page. There is little usage of the page, with a high bounce rate and low click-through rate.

These metrics result in a lower commission rate and revenue for Google. Something isn't working for users, and we believe it has to do with the current experience and design of the page.

This opportunity is perfect for a card-sort exercise. We want to understand how users categorize and organize different information on a page.

Closed versus open versus mixed

We already have a live website for this project, but it isn't performing as well as we hoped. I doubt we would remove a massive amount of features that entirely redesign every point of the experience.

Some elements exist (such as the filters) that may serve well in future iterations. However, we don't know how to organize these, and we also don't know what's missing.

I recommend a mixed-card sort. This way, we are keeping the old components for participants to organize, edit, or discard—while also allowing them to add anything we are missing.

Online versus in-person

Paper-card sorting is the traditional version of this user research method. As the name suggests, users take physical cards with topics or concepts and organize them into various piles on a large table.

The main advantage of paper card sorting is that there is no learning curve from technology.

The other option is digital-card sorting, which requires the user to perform the card sort on a website or service. The main advantage of this method is that you capture each card sorting exercise digitally, reducing the amount of work for synthesis.

For this example, we’ll do an online card sort. Some tools you can use are:

• Dscout
• Trello
• Optimal workshop
• XSort
• Miro
• Mural

Just make sure to provide detailed instructions for the participant on how to use the software. Also, leave time at the beginning of the interview for the sign-in and learning curve.

Unmoderated versus moderated

Since we're doing a mixed card sort, we'll need to recruit more participants. Ideally, you'll recruit 10 to 15 participants for open card sorting and about 30 to 50 participants for closed and hybrid sorting variations. In this case, we settle for 40 users to validate our results for the hybrid card sort.

Since 40 is a large number to recruit and host, we can mix moderated and unmoderated card sorting. We start with ten moderated card sorts, followed by unmoderated testing with the other 30 to confirm the initial results.

You can always choose to do more moderated before the unmoderated card sorts, in case you feel you don't have sufficient data.

Recruiting the right people

If you don't take the appropriate level of care in recruiting and selecting your users, you can end up redesigning your site based on the input of people who will never need to use it. We must ensure that we recruit participants who are representative of the real-life users.

I will make some assumptions here and bucket people into two major groups: "inexperienced users" of Google Shopping and "experienced users" of Google shopping. The "inexperienced users" don't frequently shop online. The users who are categorized as "experienced" shop via Google Shopping at least once a month.

You can also group by persona or marketing segment. In this case, we can recruit 20 participants from each group to total to 40.

The reason we also want to speak to those less experienced with online shopping is for acquisition purposes. We want to understand their mental models and how we could more easily capture people to purchase clothing online.

And, for retention purposes, we want to understand our experienced users' mental models. Additionally, make sure to include demographic information, such as specific age ranges, genders, or geographic locations. You want to target the demographic ranges that are most likely to use your website.

Creating the cards

If you are doing a closed- or mixed- card sort, you have to write the cards. Knowing what to include in a card sort is the tricky part. What you write on the card depends on the website or app you are looking to improve. The most important part is not to include more than forty cards in your card sort.

The best way to start creating cards is to get a site map or overall information architecture visual of your website. The site map will lay out all navigation categories, such as header navigation, primary navigation, secondary navigation (sub-categories), footer navigation, footer sub-categories, and sub-categories. You can also include filters that are on the side and not nested under any categories.

So, taking a look at Google Shopping, we see quite a lot of information, so let's tackle it one area at a time.

Header and primary navigation

The header consists of a few elements. We could include all of them:

• All
• Images
• Maps
• Shopping
• More
  • Videos
  • News
  • Books
  • Flights
  • Finance
• Settings
• Search bar

For simplicity, I would cut this portion out since the focus is on Google Shopping. Instead, I would include:

• Shopping
• Search bar

Secondary navigation

In the secondary navigation, we have some words, as well as icons. Instead of drawing out the figures, consider using the words that describe them. You can easily find this out by hovering over the image and using that text. For the secondary navigation, we would include:

• Your location
• List
• Grid
• Default
• Reviews
• Price - low to high
• Price - high to low
• Bookmarks
• Sponsored
• Relevant searches
• Previous page
• Next page

Don't forget to scroll down! There may be navigation at the bottom of the page, such as in this example.

Filters (or sub-categories)

Filters and all the options that come with them are where the card sort can get tricky. We can either include the filters and also each sub-filter or just the filter name. Remember, we don't want more than 30 to 40 cards, so we need to judge based on the number of cards we already have and will end up with.

Just the filter names:

• Price
• Style
• Gender
• Brand
• Size type
• Size
• Material
• Condition
• Shipping
• Seller

With just filter names, plus the other card, we have 24 cards. In this case, I would stop here and allow users to add clarifying cards, such as price ranges or style, during the card sort. If you want to include sub-category filters, I recommend not putting in every single brand or size.

Footer navigation

The footer navigation is at the bottom of a website. For this, we would include:

• Help
• Send feedback
• Privacy
• Terms
• Information for merchants
• Report a violation

Content

Since this is an e-commerce website, we can include actual content on the page. This step is optional. The way I would approach this is by printing out the cards and having them available. Having the content cards can help participants structure the look of the website in their minds, leading to less confusion.

Overall, we end with 30 cards of information. Since we are conducting a hybrid card sort, we will include blank cards for the participants to create their content as well.

Setting up the card sort

Usually, card sorts last 60 minutes, so plan for that amount of time. Ensure there is enough space, either physically or digitally to spread out all of the cards for the participant to see easily.

If you're conducting your card sort online, make sure you have sent the link to the participants and tell them they will need internet access.

Get someone to help you during the session to take notes and observe what you might miss. Practice the session with a colleague to make sure everything makes sense before inviting the participants.

Always record the meetings (with permission, of course)!

Moderating the card sort

Give the participant the set of cards (or have them out on a remote tool). Walk them through what the session will be like and explain what you are looking for them to do. In each scenario, mention that any cards that are confusing or they are unsure about can be put to the side.

For a closed/mixed card sort, explain that you are looking to understand how all of the items on the cards relate to each other. Tell them to group the information in a way that makes the most sense to them.

For an open card sort, tell the participant that you're trying to understand what should be in your product/service. Then explain, once they've finished naming the cards, that you want them to group them in a way that makes sense to them. Once grouped, ask the participant what they would call the grouping. Explain that you will ask for a name for each group of cards once the participant has arranged them.

Request that the participant thinks aloud during the session to fully understand their thought process behind their categorizations. If the participant struggles with thinking aloud, be sure to prompt them regularly.

Thank the participant, give them a chance to ask any questions, and provide your contact information. Be sure to mention the timeline for any incentive.

Finally, email the participant within 24 hours to thank them for their participation in the study and any incentive you offered.

Analyzing the card sort

After we have compiled all of the card sorting results, it is time for analysis and synthesis! This part, as well, can be quite complicated, especially in an open card sort. There are four steps to take when manually analyzing card sorts. Of course, digital tools, such as some of those mentioned above, can make your life easier. However, let's assume we don't have the luxury of a tool.

As a disclaimer, this is the way I have approached card sorting analysis. There are a few different ways to approach the analysis, so I encourage you to find the best technique for you.

Step 1: Identify patterns

For a closed card sort, you've already predetermined the categories. However, for an open card sort, you need to identify the most commonly suggested categories from the responses. To do this, you'll want to look for items that users frequently sort together under the same groups, and what they called those groups. Those are your categories.

Also, remember to check the cards that users frequently left in an unknown pile or discarded. If users don't know what to do with cards, those components may not belong to this section of the website.

Step 2: Create a spreadsheet or rainbow chart

There are two ways to go about this step: the less complicated version or the more complicated version. Naturally, the more complex analysis will yield better results, but it depends on your workload and timeline.

Rainbow chart

The less complicated way of approaching card sorting analysis is to create a rainbow chart. A rainbow chart visualizes the cards each participant used and shows the most commonly used cards. I recommend this method of analysis if you have little to no sub-categories and simple navigation. Read more about creating a rainbow chart here.

Spreadsheet matrix

Now, the more complicated route. I would recommend this analysis technique for this particular project, as we have sub-categories and filters. Create a spreadsheet with all the cards you wrote in the rows and each common group you identified in the column.

From your card sort's results, count how many times each card appeared under the group and add that number.

Step 3: Delete the smaller numbers

Once you've entered the data, keep the substantial groups. To quickly determine the meaningful results, set a threshold for importance. If a card appears a small number of times in a group, you could delete it. You can determine a percentage threshold as well if that is easier. For this card sort, I would remove any data under 15% or six times.

Step 4: Highlight and sort the cards

After deleting the additional data, I go through and highlight the cards that appear most often in a group. Also, I remove groups that don't have a significant number of cards (ex: no highlights). I then delete all the unhighlighted data and resort the cards. These cards would now belong to the groups they occur most often in.

Make decisions

After analyzing the results of all your card sorting sessions, you'll be in a much more informed position about the shape your IA should take and how best to structure the content hierarchy. Decide where to place content and what content needs to be changed. For instance, if many participants edited one card name (for example, style type to length), then include them in your final decision-making.

Card sorting is an excellent method of understanding users' mental models behind the organization of a website or app. This method can guide you towards making the best decisions on creating a fantastic flow and experience for your customers.

Harness the power of card sorting with Dscout

Card sorting is only one of the many features offered on Dscout. Partner with a platform that can support you across the entire research lifecycle. From recruitment and fielding to analysis and shareouts, see how our tool can help scale your research and your entire organization—schedule a demo.