Barnum and Palmer: "Tapping into Desirability in User Experience"

Barnum, C. M., & Palmer, L. A. (2011). Tapping into Desirability in User Experience. In M. Albers & B. Still (Eds.), Usability of complex information systems : evaluation of user interaction (pp. 253-280). Boca Raton FL: CRC Press.





Using MS product reaction cards to attempt to determine desirability in user experience.




Complex systems (heavier cognitive load for users) -> complex testing protocols. Usually focus on learning, interaction, reduction of error, but rarely account for how users feel about system.


You can lead a horse to water, but you can''t make it drink


ISO 9241-11-- 3 elements for gauging usability: effectiveness, efficiency, and satisfaction. Satisfaction usually forgotten, but it''s vital for continued use. So how to uncover it?


Satisfaction ratings in self-rated questionnaires: usually post-test questionnaire. Fraught with bias (not only people-pleasing; higher ratings when complete questionnaire on computer used for task). 


Post-test questionnaires in comparative studies: good for relative rates of user satisfaction. Compared five instruments: SUS (System Usability Scale), QUIS (Questionnaire for User Interface Satisfaction), CSUQ (Computer System Usability Questionnaire), Fidelity''s questionnaire (study authors'' instrument), Words (adapted from MS Product Reaction Cards). Researchers found SUS and QUIS provided most consistent responses. SUS (shortest) framed responses to site as whole. Argue that the reaction cards successful b/c provide way to tell their story of their (users'') experience. 


How (and why) Microsoft created (and used) Product Reaction Cards


Creation and Use in 2002: MS developed to try to get at desirability; standard (aka Likert) too limiting, interviews too much time. Brainstormed to find quick to administer/easy to analyze process -> desirability toolkit. Toolkit had two parts: faces questionnaire (pick face whose expression matched experience) and product reaction cards (choose descriptive words/phrases from large set of cards). Product reaction cards--select cards reflecting how they felt about product experience, narrow down to five, explain choices. Refined to develop toolkit of 118 words (60%+, %40 neg or neutral).


Application and use in 2004: for IE9, revamped for faster. Ran 10 two-hour studies w/approx 8 ppl per session/ 2 facil. Three phases:


  • I: individual measures without discussion (choose best design, rank other three choices)
  • II: individual markup w/group discussion
  • III: group consensus using product reaction cards. (group picks top three reflecting preferred design).


How others have used the product reaction cards


little info on how others have applied technique (small/anecdotal). Travis found participants tend to speak more negatively/honestly using cards vs. questionnaires, but methodology somewhat different. Rohrer instructed users to select cards to express responses to various designs. Method adapted for use in quant study. For eBay, Rohrer used paired opposites (argued helpful for creating good first impression)


Other methods used to study affect


Tractinsky, Katz, & Ikar found correlation between aesthetics and perceived usability.  Also found by Thuring & Mahlke, Phillips & Chaparro.


Our use of product reaction cards


allows way to dis/un cover how users feel about experience interacting with system; helps triangulation.


Learning to use the cards: no extant scholarship/guidelines. This article represents their analysis of their work with the cards. Started off with wrong number (55 instead of 118); changed, still finding that technique useful.


Designing methods to show the results: allows way to show clustered results in a quant manner while still semi-qual. 


Results from our studies, from simple to complex systems


  • Moving toward complexity: digital cable install kit
  • Hotel group study on loyalty program enrollment: comparing three site protoypes
  • Website for teacher professional development:
  • Network monitoring and management: time-consuming
  • A major hotel group reservation study:
  • Call center application:




product reaction cards may help to get to critical evaluation of pleasure or desirability. Notes limitations, poss acquiescence bias, time cost, shouldn''t be sole means.