Experiences with usability testing: Effects of thinking aloud and moderator presence

Sirpa Riihiaho

Research output: ThesisDoctoral ThesisMonograph

Abstract

Usability testing has become a standard method when evaluating the usability of various systems with real users. Despite this, the factors of usability testing have been given little attention in the academic forums. For example, the use of the thinking aloud method has resulted in quite conflicting effects on the users' performance in the context of usability testing, and the effects of the moderator presence have been studied very seldom in usability research. This thesis studies methods of usability testing and contextual factors in the test settings that may affect the results of a test, focusing on the effects of relaxed thinking aloud and the presence of a test moderator. It combines an extensive literature review with experiences on usability testing from 22 years covering 143 usability studies. The challenges of usability testing reported in the top academic HCI forums focus on sampling, context of use, use over time, and assessment of utility and value. The methods presented in this thesis provide some solutions to problems related to the context of use and to the assessment of utility and value. The most significant contribution of this thesis is in the experiment with thinking aloud and moderator presence as its independent variables. The results of relaxed thinking aloud show that it has no significant effect compared to silent performance on the number of usability problems the users face or in their subjective ratings, but it does slow down their performance. A significant effect of the moderator presence is found in the users' subjective rating, as users with a moderator next to them rate the system preferences significantly higher than participants performing alone. Given the benefits of having a moderator next to the user who is able to ask clarifying questions when the experiences are fresh in user's mind, the thesis still recommends this approach in formative usability testing when it is important to come up with practical redesign proposals for the development team. Thinking aloud in this experiment does not enhance the evaluators' confidence in the detected usability problems and their causes, and the test users mostly report it as an unnatural extra effort. Even so, thinking aloud gives more information about the problems to customers observing the tests, and thereby may motivate designers to make the required changes to the system. Therefore, if performance measurements are required, silent performance or classic concurrent thinking aloud with minimal interventions should be used, but in formative testing, the more explanatory relaxed thinking aloud can be used, as long as its potential effects on users' performance are kept in mind.
Translated title of the contributionKokemuksia käytettävyystestauksesta: Ääneenajattelun ja ohjaajan läsnäolon vaikutuksia
Original languageEnglish
QualificationDoctor's degree
Awarding Institution
  • Aalto University
Supervisors/Advisors
  • Nieminen, Marko, Supervising Professor
Publisher
Print ISBNs978-952-60-6226-6
Electronic ISBNs978-952-60-6227-3
Publication statusPublished - 2015
MoE publication typeG4 Doctoral dissertation (monograph)

Keywords

  • usability testing
  • thinking aloud
  • moderator presence
  • usability evaluation methods

Fingerprint Dive into the research topics of 'Experiences with usability testing: Effects of thinking aloud and moderator presence'. Together they form a unique fingerprint.

Cite this