Barnum: "Planning for Usability Testing"

Barnum, C. (2010). Planning for Usability Testing. In Usability testing essentials: ready, set...test (pp. 139-187). Burlington, MA: Morgan Kaufmann Publishers,



Defines following steps:


  • establishing the team
  • defining product issues and audience
  • setting goals and measurements for the test
  • establishing the user profile
  • selecting the tasks to include in the test
  • determining how to categorize the results
  • writing the test plan







Argues for team approach-- at least three people w/ diff viewpoints. Numbers determined by size and location of facility.


  • Defining issues/audience: may find out through surveys, response cards, etc. For test, consider:
  • how much time do you have for the test
  • how much money do you have for the test
  • where will testing be done/ under what conditions
  • who wants to know results/is sponsoring test?
  • what is best learned via usability testing/ what would be more effectively learned from other methods?


consider how usability testing built into dev cycle. argues 8 enough for users


determine testing schedule:


  • testing dates
    • #participants needed
    • length of each session
    • #days needed for all participants
  • test report/follow-up results meeting
  • pilot test
  • walkthrough (w/ or w/o user)
  • recruiting participants
  • delivery of parts of test
  • completion of test plan
  • approval of test plan


how much money?


where to test? field, lab, or remote.


argues field works best for summative evaluation (145); lab for formative eval (diagnose probs while project still developed). Address the setup (how can lab be set up?). Quiet facilitator or active intervention? Co-discovery/co-participation (2 users talk to each other while problem-solving)-- used to learn about user problem-solving approach, real-world situation supports two people, tends to promote more comments than TAP-- however, need 2x many users, minimze collection of quant data, 1 person may dominate; retrospective recall.


measurements: what do you want to learn from test, how do you want to measure results?


want to operationalize goals (define in specific, measurable terms). Goals are based on tasks but tasks are not goals.


p 156: Progression from General Goal to Concerns to usability goal


General Goal: Create input forms that users will want to use.
Specific concern: Will the users want to work with the input forms, or will they avoid the forms?
Specific concern: Will users attempt to update the data records by opening the form in a data sheet view instead of in a form view, effectively bypassing the input fonn?
Quantitative usability goal: Each user will switch to the database view of a form no more than once per sessian
General Goal: Design input forms through which users can navigate quickly and easily.
General concern: Will users be able to navigate through the forms?
Quantitative usability goals:
1. The user will be able to locate the correct form for the task in 2 minutes or less.
2. The user will be able to locate a blank form to enter a new record in 2 minutes
or less.
3. Time needed to locate the correct form should be less for the last task than it
was for the first attempted task by each user. iGeneral goal Create input form that users will prefer over previous procedure.
Specific concern: Wil lusers find greater satisfaction inusing the new input form vs. the previous
input procedure?
Qualitative usability goal: Users will rank the new form with at least a 4 (on a 5-point scale, with 5 being the
best) when asked their satisfaction level with the form.


Quant tells you something is wrong; qual tells you what is wrong from user''s POV.


User profile-- Chauncey Wilson recommendations for user profile:


Motivation-how motivated are users to use the product?
• Typing skills-ccritical for some jobs.
• Task experience-familiarity with the task that the product supports.
• Turnover rates-high turnover rates would indicate that ease of learning issues is critical
• General education.
• Learning style.
• Computer background.
• Software applications used.
• Expected training support for the product.
• Age ranges.
• Stress level when performing tasks.
• Domain (subject matter) knowledge.
• Number of people involved in performing the tasks.
• Job category.
• Level of automation.
• Attitude toward the job.
• Location of use (home, office).
• Physical characteristics.
• Disabilities-what percentage of the user population has disabilities that could affect product use?
• Number of people in user population with English as a second language.


Subgroup considerations:


• Length of time using the product
• Number of times per week they use a product
• Number of hours per week they use a product
• Specific tasks they perform with the product (this could be a list or it could be open-ended to see how they respond)


Points to lack of research as to gender and usability


Selecting tasks-- based on operationalized goals/needs of users identified.


Tasks are frequently determined by using the following criteria (161):
• First impressions (look and feel of the product) .
• First tasks (so important in fixing in users'' minds whether they WIll consider
the product easy or difficult to use)
• Tasks most frequently performed
• Critical tasks (even if perfonned less frequently) . .
• Specific problem areas (typically identified by sponsor or heuristic evaluation)
• New tasks added to a product or changes made from an earlier version of
the product (including changes made after an earlier usability test of a


How to categorize (working with established set of categories). Bottom up approach or top down?


Importance of test plan: serves as test blueprint, main communication vehicle, describes or implies required sources, provides focal point for test/milestone for project.


See setup on 171 for format of test plan.