Faulkner: "Beyond the five-user assumption: Benefits of increased sample sizes in usability testing"

Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, 35(3), 379-383.

 

 


 

 

again takes on the 5 user assumption. Argues need for larger samples of multilevel users and a way to make data accessible to non-statisticians to demonstrate risks of 5 user assumption.

 

Background: Nielsen''s and Virzi''s work. Cites Spool and Schroeder.

 

Methods: Faulkner designed study of 60 users/ sets of different sizes. Usability test of web-based employee time sheet. Users classified as e.g. novice/novice, expert/novice (know comps but not app), expert/expert Measured time to completion, user deviation. Calculated e.g. SD.

 

Results: Found how statistically picking from random samples that with 5 users the confidence interval was 18.5% (eeek!).

 

Discussion: Findings demonstrate possible false confidence that could come out of 5 user sample.

 

Conclusion: Need to understand 5 user assumption implications and origins. Answer to how many: it depends. Test group must rep target population. Want to *really* save money? Don''t skimp on the user testing-- it will cost much more if problems not found.