Crowdsourcing in Computing Education Research: Case Amazon MTurk

Arto Hellas*, Albina Zavgorodniaia, Juha Sorva

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

120 Downloads (Pure)


Crowdsourcing platforms such as Amazon MTurk provide access to a human workforce that can be given tasks to complete online for a fee. In this article, we review studies in computing education research (CER) that rely on crowdsourcing; we also describe our own experiences of using Amazon MTurk for a CER study. We discuss challenges in recruiting workers with specific backgrounds—such as no programming experience—and considerations in filtering out unreliable research participants. Combining recommendations from the literature with the lessons that we learned whilst conducting our study, we synthesize advice for researchers in CER who are considering crowdsourcing. In our case study, we did not find widespread foul play by crowdsourced workers and, overall, our experiences and the literature suggest that crowdsourced CER is feasible. It is, however, uncertain to what extent crowdsourced data can produce answers that apply to specific educational contexts. More research is needed before definitive conclusions can be drawn about the validity and generalizability of crowdsourced CER.
Original languageEnglish
Title of host publicationKoli Calling '20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research
ISBN (Electronic)978-1-4503-8921-1
Publication statusPublished - 19 Nov 2020
MoE publication typeA4 Conference publication
EventKoli Calling - International Conference on Computing Education Research - Virtual, Online
Duration: 19 Nov 202022 Nov 2020


ConferenceKoli Calling - International Conference on Computing Education Research
CityVirtual, Online
Internet address


  • Crowdsourcing
  • CER


Dive into the research topics of 'Crowdsourcing in Computing Education Research: Case Amazon MTurk'. Together they form a unique fingerprint.

Cite this