Barnum: "Preparing for Usability Testing"

Barnum, C. (2010). Preparing for Usability Testing. In Usability testing essentials: ready, set...test (pp. 188-229). Burlington, MA: Morgan Kaufmann Publishers,



Documents for usability test:


  • screening questionnaire(s)
  • pre-test questionnaire
  • post-task questionnaires
  • post-test questionnaire
  • consent/release forms


Creating the Screening Questionnaire


Devise questions identifying characteristics of user


recruit at least three (pref 4 or 5 users) for each profile


Usually focus on proficiency questions-- how often, how long, how much. Don''t ask users to ID own levels of knowledge/proficiency. Instead, open-ended questions or ranges. See samples on 190-91


Recruiting Participants


budget at least 2 hours per participant (4-15 calls per user)


Recruit Directly: customer lists, potential customers, user groups/prof organizations, ads, qual friends. Telephone most time-consuming but yields most direct/fastest results. Don''t recruit people at work. Should offer some incentive.


Recruiting Indirectly Through Agencies: Market research firms tend to produce better participants. Supply your screening questionnaire. Regular use of same agency better.


Scheduling Participants: Calling works better than email, but also email a reminder. Anticipate 10% no-shows. Need back-up plan:


  • Double book
  • Have a backup nearby
  • Schedule an extra person or two


Further Screening on Arrival: can use same questionnaire or create additional (aka pre-test) questionnaire. Need to decide what to do if user doesn''t match profile.


Creating Scenarios


Avoid creating steps/instructions (focus on process); avoid using language of the product (unless you are sure that terminology is not the issue). For each scenario, create post-task questionnaire for immediate feedback.


If you want to test documentation, instruct users to work as normal and observe whether they consult documentation or direct users to documentation. 


Creating Post-Task and Post-Test Questionnaires


Likert scale; or neutral statement (how difficult? very easy, somewhat easy, neither easy nor difficult...); or rate level of satisfaction; or list category/present topic.


Post-test frequently asks for qual and quant responses.


Defining Team Members'' Roles


Minimum three people:


  • facilitator: (direct interaction with participant) aka briefer/ debriefs.
    • creates checklist of tasks/reminders 
    • scripts/outlines
    • consent/release forms (nondisclosure; cosent)
  • Logger: fastest typist
    • observation form
  • Camera/video operator: equipment




  • narrator: assists logger in capturing findings
  • help desk operator: person most familiar with product
  • test administrator: manages all phases of projects (often combined with facilitator)


What makes a good moderator?: relaxed; practices neutral "prods"; watch vid tape of self; avoid use of loaded words (test, subjects); some positive reaction to participation; don''t talk about participants in public spaces; gracefully decline inappropriate participants; run pilot session; schedule enough time between sessions; protects privacy; has checklist; uses script; dresses sim to participants; strategies for getting feedback from shy participants; knows when to end task; may give tips on using software more effectively (at end of study).


Conducting the Walkthrough and the Pilot


Walkthrough: chance to test equipment, scenarios, etc. Recruit a tolerant user-- can deal with mistakes and confusion. Purpose is to find flaws in the plan. Tech rehearsal!


Pilot: 2-3 days before full testing begins. Chance to practice approach, etc. Pilot participant must match user profile.