How many users makes a good usability test?

Article published on December 29, 2005 under Usability

Every usability practitioner, early or later, run into a simple question: how many users I need to conduct a good usability test? Jacob Nielsen argue in his alertbox that five users are enough to detect 80% of all usability problems. On the other hand Christine Perfetti and Lori Landesman claims that eight people are not sufficient to discover all usability problems. So what is the answer to our question?

Before giving an answer to our question we have to take in considerations some factors that influence the number of people that had to be hired in order to conduct usability tests:

  • Project budget
  • Project timeline
  • Project complexity

Project budget

The biggest obstacle for usability is represented by project budget. Here, in Italy, companies does not invest on usability and all related subjects are treated as optional services in estimates. Clients infact prefer get discounts in the final price by cutting off usability costs (including user testing) or more frequent investing available budget in other services like home page flash animations. These prejudices toward usability influence in the negative the final product and when clients realize of the mistake they made, is too late to find solutions without investing more money. So there's no budget to hire people and conduct usability tests.

Project timeline

Clients believe that good web site can be done in a week and insists to get it published in five days: one day to get material, one day to prepare layout, two days to develop pages and last day to make funcionality test and deploy site. There's no option to consider usability studies in such timeline. So there's no time to hire people and conduct usability tests.

Project complexity

To analyze this factor I will take as an example web sites projects that can be divided into four types (naturally this is a simplified categorization):

  • small: web sites that goes from a simple waiting page to 10 static pages
  • medium: web sites that exceed 10 static pages or includes few dynamic contents and special functionalities
  • large: web sites based on dynamic contents and include a lot of special functionalities
  • web applications

Logically usability analysis is useless in small size web site due to their small complexity and most important for their reduced budget. In medium size web sites and in web applications usability analysis can be done either using an heuristic evaluation or usability test with no more than five users. Naturally, if funds and time are sufficient is better use both options. In large web sites depending on budget (timelines in this kind of web sites in generally more relaxed) is better conduct heuristic evaluation and usability test with no more than ten users.

The answer

So depending on the external factors the best result can be achieved making usability test with no more than ten people in most cases closer to five. Moreover is better conduct usability tests integrating them with an iterative approach ideally following this procedure:

  • Conduct a heuristic evaluation in prototyping phase so biggest usability issues can be early resolved
  • Conduct a usability test with four users at the end of design process in order to correct design usability problems and collect feedback on interfaces
  • Conduct a comprehensive usability test with four users at the end of development phase in order to correct interaction usability problems
  • Conduct a usability test with two users at the end of the deploy process in order to verify that all usability issues have been corrected and also to gather feedback that can be used as recommendations for future version.