To put it simply: there are three main things you need to focus on when planning and running user tests:
- Recruiting representative participants
- Defining a representative task to solve with the product
- Stepping back as much as possible and letting the participant do the talking
The first item in this list emphasizes that running user tests without the appropriate participants is impossible. While recruiting some kind of participants isn’t usually a problem, you should keep in mind several general conditions that influence the recruiting process.
Recruiting representative Users
Anybody who runs UX and usability tests regularly will quickly notice that the recruitment of participants is the basis for your research. Having a structured recruitment process including channels to address users, screening questions and appropriate incentives increases your efficiency considerably. If you manage to recruit efficiently, you’ll be able to invest the resources you saved during the recruitment process into running a bigger number of tests. On top of that, you’ll be able to choose targeted participants more purposeful, which will positively influence the quality of your collected data.
In the end, recruiting fitting participants always has its price: the average price you need to pay agencies for recruiting your participants varies widely with the number of characteristics you defined for your participants. If you have employees who do the recruiting for your company you need to consider their worktime spent. You should assume that it will take them one to two hours to find a fitting candidate.
Criteria for choosing representative participants
In the recruiting process, it is extremely important that you ask real users of the product you want to test (or if it’s a new product similar products) to participate in your research. If you can’t convey actual users to take part, make sure your participants match your actual users as closely as possible. If they don’t, discovering errors in your system and evaluation of the benefits of your products during realistic tasks becomes unnecessarily hard.
To identify the part of the population that is similar enough to your participants to provide you with valuable insights, you need to be informed about their characteristics. Recruiters should figure out what the product is supposed to do and what exactly is the area of application. This is the fastest way to gain a general idea of the supposed user group. as well as obstacles that may arise during the recruiting process. To limit the target group further, recruiters should question different departments that possess information about the target groups characteristics. E.g. developers, the marketing team, employees working in customer support… In certain cases, it can be helpful to take a closer look at the user base of a competing product.
Number of Participants
The number of participants you need to recruit is highly dependent on the research method you are using as well as your study’s goal. Studies that can be evaluated using statistical analysis methods usually need participant numbers in the double-digits. A rule of thumb is that you can start calculating meaningful statistical significances starting at 20 participants per experimental condition. Generally speaking, you can assume that your sample will be bigger if you use quantitative methods.
If you aim to do qualitative or explorative research, it doesn’t necessarily make sense to test using as many participants as possible. Depending on the kind of test you use, five participants are plenty as they should be able to uncover 80% of the occuring usability problems in a product. With every additional participant, the added benefit decreases, since they tend to repeatedly uncover the same obvious flaws. Because of this, stopping your usability tests after examining five participants is reasonable. If necessary you can run additional tests with the next iteration of your product.
If you’re planning to run multiple tests with one product or even different iterations of a product, you should always use different groups of participants. If you’re testing different tasks or you know that your test will take over an hour for some other reason, you should consider splitting your test into multiple smaller test sessions and running them with different groups of participants. That way you can avoid survey fatigue, which will lead to a decrease in the quality of your data.
Once you made a list of all relevant characteristics of your target group, the next step is thinking about what percentage of your sample should have what characteristic. Regardless of which channel you use to recruit your participants, you’ll still need to filter out the part that resembles your target group close enough. Doing this you should choose criteria that are specific enough to portray your target group but not so specific that you’ll run into trouble finding participants at all.
Of course, you should ask your participants the screening questions before you allow them to sign up for your test. For example: assuming you use some kind of online form to enable participants to sign up for your test, you could only display said form after having them fill a questionnaire containing your screeners. Ideally, you should ask the questions that will filter out the most participants first and stop the process as soon as participants answer a question in a way that shows they don’t fit your target group. That way people who don’t match your target group don’t have to go through the hassle of filling out the whole questionnaire just to be excluded from taking part in your test.
Especially if you’re employing remote tests and paying your participants, there will be a great number of people who want to take part in your test whether they match your target group or not. Because of this, you should try to phrase your questions in a way that doesn’t make it obvious to participants which characteristics you’re looking for and which you are excluding.
Recruiting participants for your research without offering them some kind of reimbursement is nearly impossible. That’s why lots of laboratory studies use so-called incentives to reduce the lack of participants. If you’re testing remotely you need incentives to motivate users to even take part in your study at all. When deciding what to use as an incentive you can choose every kind of reimbursement for the time invested that is attractive to your participants. Raffles for higher-priced items, vouchers, merchandise or of course money are some of the most used incentives.
If you participants need to take a day off of work to take part in your study need to be compensated in a way that results in them not ending up with a loss. That way incentives can add up quickly. Incentives for participants that take part in your tests remotely are considerably cheaper. Still, you need to keep in mind that incentives need to be higher for target groups that are hard to reach and will also differ depending on their socio-economical status.
Lastly, you need to be more generous when compensating participants for studies that take longer or contain more difficult tasks. The last thing to keep in mind is that you should always thank your participants for their time-either directly, in a follow-up mail or both-whether you pay them or don’t.
Channels for recruiting Participants
- Recruitment agencies and services: working with agencies, panels or other recruiting services is especially useful if you want to test with a very broad target group or under time constraints.
- Lists: if you already have a mailing list containing people with the characteristics you’re looking for you can use those to recruit participants. These lists can be lists of existing users of a previous version of the product you’re testing. Other groups of people who are close to your company can be recruited via pop-ups on your website, making announcements on your company’s social media pages or via your newsletter.
- Additional channels: other ways to address external participants are ads in search engines and social media channels as well as (online) special interest groups that match your target group.
Even if you do your best to recruit the right participants and offer them attractive incentives you have to anticipate that about ten percent of your participants will not provide you with usable data. While the main reason for this problem in laboratory settings is the previously named lack of participants the problem arises in remote tests as well because of a great number of people who don’t finish your test. This is the reason why you should either recruit additional participants from the get-go or as soon as you realize that you won’t be able to collect enough data.
Once you collected data from your participants, consider.ly will help you to manage, store and evaluate your data.
Start your 30 days free trial now!