Boren and Ramey: "Thinking Aloud: Reconciling Theory and Practice"

Boren, M. T., & Ramey, J. (2000). Thinking Aloud: Reconciling Theory and Practice. IEEE Transactions on Professional Communication, 43(3), 261-278.

 

 

 


 

 

 

States no detailed description of theoretically motivated rules of practice for TAP.

 

References Ericsson and Simon as the first "source," but history behind

 

W. James and Wundt, used verbalization to investigate in psych. Dropped out of fashion; made comeback in 60s.

 

Ericsson and Simon (cognitive psych) defined three levels of decreasingly reliable verbalization:

 

  • Level 1 Verbalizations are those that need not be transformed before being verbalized during
    task performance
  • Level 2 Verbalizations are those that must be transformed before being verbalized during task performance
  • Level 3 Verbalizations are those that require additional cognitive processing beyond that required for task performance or verbalization (E&S argue against this kind of data).

 

Some kinds of verbalization cannot be considered data b/c they raise probs with interpretation and analysis (stream of consciousness, daydreams, feelings) (262).

 

Distills points from E&S for usability:

 

  • collect and analyze only "hard" data
  • Give detailed instructions for thinking aloud.
  • Remind em to think aloud
  • Otherwise, don''t intervene

 

Theory and Practice Are Out of Sync

 

Notes that wide range of variation; if E&S consistently applied, wouldn''t be so. Boren and Ramey conducted field research as to how TAP collected/analyzed.

 

Noted discrepancies:

 

  • usability practitioners often don''t give Think-Aloud instructions in the prescribe manner (264) (e.g., not differentiating between think-aloud and explanation).
  • UPs don''t often give reminders in prescribed manner (just ask to keep talking). Discrepancies among prompts (E&S ''imply'' predetermined interval 15-60 secs). Usability no reg prompting interval. Often contextual
  • UPs often intervene in theoretically inconsistent ways (265); some ''help,'' req to clarify a comment
  • UPs often don''t use verbalizations as ''hard'' data--instead, qual.

 

Sources of Disparity Between Theory and Practice

 

most UPs don''t actually subscribe to model in practice. Perhaps think procedures more complex than necessary, or that don''t provide collectable data.

 

Reconciling Theory and Practice

 

  • Stop collecting verbal protocols/ just use observable performance
  • strictly apply E&S''s theory
  • Explore theoretical alternatives

 

Proposal: Explore Speech Communication as a Theoretical Alternative

 

  • disagree that listener is entirely passive
  • analyze speech genres

 

Proposed Areas of Investigation for Speech Communication and Usability Testing

 

  • First, we describe how usability practitioners could use speech communication theory to help set the stage for productive interaction.
  • Second, we suggest modifications to data-collection methods which would take into account the social nature of speech, but which work toward the same goal as Ericsson and Simon’s procedures: eliciting a verbal report that is as undirected, undisturbed, and constant as possible.
  • Third, we consider interventions often required by the contingencies of usability testing, but for which Ericsson and Simon give little guidance.
  • Finally, we discuss proactive interventions that elicit information that is not allowable
    under Ericsson and Simon’s model. (268)

 

To set stage consider:

 

  • the product being tested is the subject
  • the participant is the work domain expert and primary speaker
  • the UP is the learner and primary listener

 

For facilitator:

 

  • Keep nature of speech to keep verbal reports undirected, undisturbed, and constant.
  • Use acknowledgment tokens continuously
  • Choice of token affects available responses. (go for mmhmm or uhuh) (269)
  • Frequency of acknowledgment
  • Arguments against Continuous acknowledgment
  • Remind participant to keep talking

 

Handling the Contingencies of Usability Testing that REQUIRE Interaction Between Usaibility Practitioner and Participant.

 

  • Trouble with maintaining monolog: if system crashes, suspend or quietly fix
  • System crash, bug, prototype incomplete: reassure user did not cause problem/issue with product.
  • participant avoids vital functionality: user thinks done, but not (conclude with statement, "you''ll be done when...") (argues for further research): user sidesteps functionality of interest (might ask to try a different way); participant stuck (273). Stand pat or encourage user to continue-- what is more important to process?
  • can''t deal with participant questions (repeated or not in task list). Turn question back to user. Chatty users-- may use leading question (now you''re working on...)
  • proactively eliciting additional info: again, argues how rules of speech communication better than E&S. clarify unclear comment; turn question back on user, etc.

 

Conclusion

 

issue may be that some think usability tests should be more like interviews. NO!