Faulkner: "Beyond the five-user assumption: Benefits of increased sample sizes in usability testing"
- Category: Annotations
- Hits: 1631
Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, 35(3), 379-383.
again takes on the 5 user assumption. Argues need for larger samples of multilevel users and a way to make data accessible to non-statisticians to demonstrate risks of 5 user assumption.
Background: Nielsen''s and Virzi''s work. Cites Spool and Schroeder.
Methods: Faulkner designed study of 60 users/ sets of different sizes. Usability test of web-based employee time sheet. Users classified as e.g. novice/novice, expert/novice (know comps but not app), expert/expert Measured time to completion, user deviation. Calculated e.g. SD.
Results: Found how statistically picking from random samples that with 5 users the confidence interval was 18.5% (eeek!).
Discussion: Findings demonstrate possible false confidence that could come out of 5 user sample.
Conclusion: Need to understand 5 user assumption implications and origins. Answer to how many: it depends. Test group must rep target population. Want to *really* save money? Don''t skimp on the user testing-- it will cost much more if problems not found.