How to Recruit Participants for Usability Testing

by Mara Weingardt | May 24, 2019 | General

| 8 min read

When conducting user research, you will need to recruit participants that attend your usability tests, interviews, or other research methods. Recruiting participants is the basis for your research, as they provide you with the data to draw conclusions on. However, how to recruit participants properly is a research area for itself (Nielsen even wrote a 190-page report about it).

How do you even recruit the right participants? How many should you invite? Should you pay them and how much is appropriate? In the following, we’ll give you an overview of this topic to prepare you for your next user testing.

How do I select representative participants?

The most important part of your user tests are: The users! That is why it is important to take a closer look at who you actually want to recruit.

Usually, most of the usability issues can be found by any user. But depending on your product there may be some peculiarities that are only present in your specific target audience.

The better your test participants match your actual target audience the better your test results will predict how your product will be perceived out in the wild. If it’s a new product you can look for similar products and address their respective target audiences.

To gain a general idea of the supposed user group, try describing what the product is supposed to do. This will allow you to define the exact area of application. This is the fastest way to gain a general idea of the supposed user group. To specify the target group further, you should question different departments that possess information about the target group’s characteristics, e.g., developers, the marketing team, or employees working in customer support. In certain cases, it can be helpful to take a closer look at the user base of a competing product.

Either way, you should have a clear picture of the characteristics of your target audience, e.g., by working out a persona. This is the basis to define screening questions and also communicate your request to external recruiters.

If you can’t convey actual users to take part, make sure your participants match your actual users as closely as possible. Only real users or similar participants can discover errors in your system and discover the benefits of your product. But as always it is a trade-off between time, money, and participant quality.

How many participants should I invite for my study?

The number of participants you need to recruit for your UX research depends on two things: The used research method and on your study’s goal.

Studies that can be evaluated using statistical analysis methods usually need participant numbers in the double-digits (10 or higher). As a rule of thumb is that you can start calculating meaningful statistical significances starting at 20 participants per experimental condition. Generally speaking, you can assume that your sample size needs to be bigger if you use quantitative methods.

Why 5 participants is the magic number of usability testing

If you aim to do qualitative or explorative research, it doesn’t make sense to test as many participants as possible. Usually, five participants are sufficient as they should uncover 85% of the occurring usability problems in a product. With every additional participant, the added benefit decreases since they tend to repeatedly uncover the same obvious flaws. If necessary, you can run additional tests in another study or with the next iteration of your product.

Smaller tasks for better data

If you’re planning to run multiple tests with one product, you should always use different groups of participants. If the usability test will take more than an hour, you should consider dividing the testing session into smaller tasks. And then test them with different groups of participants.

Obviously, you will then need more participants. But rather include more participants in your study, than trying to squeeze out every last bit of information from them.

By keeping the tasks small you avoid survey fatigue (i.e., participants becoming bored or uninterested in your study). That would lead to a quality decrease in your collected data.

How should I screen my participants?

To ensure that the participants match your target audience and no other participants sneak in, you need to screen them via a questionnaire (a.k.a. a screener). This questionnaire asks specific traits of your target audience and only lets people pass if they match your criteria.

Define the sample for your study

First, create a list of all relevant characteristics of your target group. Next, think about how many respondents in your sample should have which characteristics (e.g., you’d like to test an equal amount of women and men).

Regardless of which channel you use to recruit your participants, you still need to narrow down the part that actually represents your target audience before your study starts. For doing this, choose criteria (demographical, behavioral, or psychographical) that are specific enough to portray your target group. On the other hand, don’t choose the criteria too specifically, since then it may be difficult to find participants at all.

You shall not pass!

Obviously, you should ask your participants the screening questions before you allow them to sign up for your user test. E.g. you can use some kind of online form (like Google Forms) to let participants sign up for your test (after filling out your screener).

In your screener, you should ask the questions that will filter out the most participants first. Stop the screening process as soon as participants answer a question in a way that shows they don’t fit your target group. That way people don’t have to go through the hassle of filling out the whole questionnaire just to be excluded from taking part in your test.

Don’t make it obvious for the participants

Especially if you’re employing remote tests and paying your participants, there will be a great number of people who are up to taking part in your test (whether they match your target group or not). Because of this, you should try to phrase your questions in a way that doesn’t make it obvious to participants which characteristics you’re looking for.

If you want to include or exclude a specific trait or industry, do not ask for that in specific. Instead, give more options to choose from. For example, if you’d like to include participants that work in automotive, do not ask: “Are your working in the automotive sector? Yes/No”. Instead ask: “In which Sector are you working? (choose one): Finance, Service, Automotive, …”.

Check for the ability to articulate

Ask one open-ended question to see how your participants can articulate themselves. That doesn’t necessarily have to do with your research goal. Let them describe their favorite web page or ask them about their favorite movie and why they like it. Note that this, of course, can also bias your research. Maybe your target audience is rather introverted. Then this question would sort out a lot of them. Nevertheless, it is usually better to have a test person in the test who can open up and articulate her experiences and problems.

How should I incentivize participants?

Usually, participants in research studies are incentivized simply by a certain amount of money. That should cover their travel expenditure (normally when they need to come to your lab) and reward their time. Consequently, lab studies will cost more than remote tests and longer studies require higher incentives. Also, anticipate that high earners will expect more money than low earners.

Motivate participants the right way

Paying participants for their attendance is not without critique in the research community and might even raise ethics questions. Make sure that it’s clear that participants can always withdraw from the research and still receive payment. No persuasion or pressure of any kind should be put on your participants.

When deciding what to use as an incentive you can choose every kind of compensation for the time invested. As long as it’s attractive to your participants. Vouchers and of course money are some of the most used incentives. Otherwise, depending on your target group, you might find better incentives like giving away merchandise, donating to charity, planting a tree, and so on.

How much is enough? It depends…

Sometimes, participants need to take a day off to take part in your study. They, of course, should be compensated in a way that results in them not ending up with a loss. That way incentives can add up quickly (and we all know that research budgets are limited).

Incentives for participants that take part in your tests remotely are considerably cheaper. Still, you need to keep in mind that incentives need to be higher for target groups that are hard to reach and will depend on their socio-economic status.

Also, you need to be more generous when compensating participants for studies that take longer or contain more difficult tasks.

Lastly, besides monetary compensation, always thank your participants for their time and input!

Channels for recruiting participants

Recruitment agencies and services: Working with agencies, panels, or other recruiting services is especially useful if you want to test with a very broad target group or under time constraints.
Email lists: Often you already have a mailing list containing people with the characteristics you’re looking for. For example, a list of existing users of a similar or previous version of the product you’re testing.
People who are close to your company can be recruited via pop-ups on your website, making announcements on your company’s social media pages, or via your newsletter.
Other ways to address external participants are online ads in search engines and social media channels. Also, you look for participants in (online) special interest groups, like Facebook groups or Slack communities, that are tied to your target audience.

Even if you do your best to recruit the right participants and offer them attractive incentives you have to anticipate that about 10% of your participants will not provide you with usable data. Speaking from experience, this problem especially arises in remote tests, where a great number of people won’t finish your test. Therefore, you should recruit additional participants from the get-go. Or at least as soon as you realize that you won’t be able to collect enough data.

Streamline your participant recruitment

Having a structured recruitment process makes your research more efficient. If you recruit efficiently, you’ll be able to invest the time and resources you saved into running more tests and choosing participants more selectively. Both will (to some degree) positively influence the quality of your collected data.

Regardless of whether you’re recruiting by yourself or via an agency, don’t forget to document the whole process. This gives you a playbook for the next time and also helps colleagues to support you with it.

Recruiting participants by yourself

Recruiting participants by yourself can be quite challenging, depending on your target audience.

If you have people using your product, you can just drop them an email and ask them if they would like to take part in your research. If you want to target non-users or potential users of your product, you have to investigate where to find them. For easy to reach target audiences, you can go online, place some Facebook ads, or ask friends and family. If you want to have more specialized participants it can help to head for special interest forums or online communities that reflect your target audience.

After you found your test participants you should ask them for consent to put them on your own internal participant list. This way you can build up your very own panel of participants to contact if you want to do research again. However, keep in mind that you will scare them off if you contact them too often.

Sometimes there are hard to reach participants (like dentists or Lamborghini drivers) and depending on your product it can be more cost- and time-efficient to use a specialized recruiting service.

Recruiting participants via an agency

You want to recruit participants via a specialized agency and let others do the hard work. Great! But even then, there are some pitfalls to avoid and processes to streamline.

Before contacting an agency, make sure you have an exact description of your desired test user. Then think about with which questions you could target these people. You don’t have to use a questionnaire tool for that, for now, it is enough if you just write it up in a document.

Then highlight the desired answers that your participants should give, in order to qualify for your research. You can also give quotas, to signal the panel how much of each participant group you want in your research (for example if you want 30% of your participants to be male). This will be a rather big one-time effort, but it will help you in the future for similar research tasks. Save this questionnaire as a template and just change the parameters for future recruiting if needed.

After the questionnaire is done, you can email it to an agency or a panel. They will let you know how much the recruitment will cost. More and more agencies have online calculators that let you choose your desired audience without sending them a screener. You should always request several panels for a quote and compare their prices, as they often are specialized in different fields.

After every research, document who provided you with which target audience to which price (and note their contact email address). When you are ready to recruit again, you can simply send requests to the respective agencies. Luckily, you won’t have to search much longer. Furthermore, this will help you in better planning your budget.