Barnum: "Preparing for Usability Testing"

Barnum, C. (2010). Preparing for Usability Testing. In Usability testing essentials: ready, set...test (pp. 188-229). Burlington, MA: Morgan Kaufmann Publishers,

 


 

Documents for usability test:

 

  • screening questionnaire(s)
  • pre-test questionnaire
  • post-task questionnaires
  • post-test questionnaire
  • consent/release forms

 

Creating the Screening Questionnaire

 

Devise questions identifying characteristics of user

 

recruit at least three (pref 4 or 5 users) for each profile

 

Usually focus on proficiency questions-- how often, how long, how much. Don''t ask users to ID own levels of knowledge/proficiency. Instead, open-ended questions or ranges. See samples on 190-91

 

Recruiting Participants

 

budget at least 2 hours per participant (4-15 calls per user)

 

Recruit Directly: customer lists, potential customers, user groups/prof organizations, ads, qual friends. Telephone most time-consuming but yields most direct/fastest results. Don''t recruit people at work. Should offer some incentive.

 

Recruiting Indirectly Through Agencies: Market research firms tend to produce better participants. Supply your screening questionnaire. Regular use of same agency better.

 

Scheduling Participants: Calling works better than email, but also email a reminder. Anticipate 10% no-shows. Need back-up plan:

 

  • Double book
  • Have a backup nearby
  • Schedule an extra person or two

 

Further Screening on Arrival: can use same questionnaire or create additional (aka pre-test) questionnaire. Need to decide what to do if user doesn''t match profile.

 

Creating Scenarios

 

Avoid creating steps/instructions (focus on process); avoid using language of the product (unless you are sure that terminology is not the issue). For each scenario, create post-task questionnaire for immediate feedback.

 

If you want to test documentation, instruct users to work as normal and observe whether they consult documentation or direct users to documentation. 

 

Creating Post-Task and Post-Test Questionnaires

 

Likert scale; or neutral statement (how difficult? very easy, somewhat easy, neither easy nor difficult...); or rate level of satisfaction; or list category/present topic.

 

Post-test frequently asks for qual and quant responses.

 

Defining Team Members'' Roles

 

Minimum three people:

 

  • facilitator: (direct interaction with participant) aka briefer/ debriefs.
    • creates checklist of tasks/reminders 
    • scripts/outlines
    • consent/release forms (nondisclosure; cosent)
  • Logger: fastest typist
    • observation form
  • Camera/video operator: equipment

 

also:

 

  • narrator: assists logger in capturing findings
  • help desk operator: person most familiar with product
  • test administrator: manages all phases of projects (often combined with facilitator)

 

What makes a good moderator?: relaxed; practices neutral "prods"; watch vid tape of self; avoid use of loaded words (test, subjects); some positive reaction to participation; don''t talk about participants in public spaces; gracefully decline inappropriate participants; run pilot session; schedule enough time between sessions; protects privacy; has checklist; uses script; dresses sim to participants; strategies for getting feedback from shy participants; knows when to end task; may give tips on using software more effectively (at end of study).

 

Conducting the Walkthrough and the Pilot

 

Walkthrough: chance to test equipment, scenarios, etc. Recruit a tolerant user-- can deal with mistakes and confusion. Purpose is to find flaws in the plan. Tech rehearsal!

 

Pilot: 2-3 days before full testing begins. Chance to practice approach, etc. Pilot participant must match user profile.