Cooke: "Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach"

Cooke, L. (2010). Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach. IEEE Transactions on Professional Communication, 53(3), 202-215. doi:10.1109/TPC.2010.2052859

 


 

Introduction

 

discusses divergent backgrounds of usability researchers in cog sci/ HCI vs tc. Tc b/g rhet analy, stat analy, content, discourse, viz, etc. analy + audience analysis-- highly aware of rhetorical situations.

 

Purpose of essay is to assess how concurrent think-aloud protocol (CTA) treated in the disciplines.

 

Ericsson and Simon popularized in 80s. Covers three levels of verbalizations.

 

Arguments against CTA in cog psych: doesn''t record unconscious/immediate, but processed thoughts; changed sequence of thought process; act of thinking aloud interfered with cog processing. Cites Boren and Ramey.

 

Notes use of CTA in comp theory (Flower and Hayes; writer experiences).

 

CTA and Usability Testing

 

started in 1990s but used inconsistently; thereby criticized for lack of rigor. Points out gaps in understanding CTA and usability testing.

 

Three goals of study:

 

  • RQ1. To what degree are the statements users produce during CTA accurate?
    RQ2. What content categories do CTA verbalizations include?
    RQ3. What do users’ eye movements reveal about their behavior when they are silent or using verbal fillers? (204)

 

Methods

 

UW-- 4 days-- 5W 5M 22-41. All had participated in CTA testing before. Used Wash state dept of licensing site. Four tasks. Used eye-tracking software. Major issues head stillness for device/ loss of calibration. Participants not told what studying was measuring other than general usability.

 

Diagram of data processed on 207. Developed categorizations of verbalizations (reading, procedure, observation, etc.). Research addressed RQ1 and RQ2, but haven''t had chance as yet to delve into RQ3.

 

Results and Discussion

 

Verbalizations verified by eye movement 80% of time (209). Other 20% discrepancy could be related to issues of eye tracking software lag, visual processing vs. verbalization, etc.

 

Verbalizations coded for content analysis. Majority (58%) reading; 19% procedure (210). Reading and procedure high in Bowers and Snyder, but priorities flip-flopped. Observation 10%; users used to orient selves visually to webpage. 

 

Verbal fillers/silence 16% of time across tasks. Seem to fill assessment function.

 

Conclusion

 

CTA yields low percentage of explanation verbalizations.