Crowdsourcing in Computing Education Research: Case Amazon MTurk

Arto Hellas*, Albina Zavgorodniaia, Juha Sorva

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

Abstract

Crowdsourcing platforms such as Amazon MTurk provide access to a human workforce that can be given tasks to complete online for a fee. In this article, we review studies in computing education research (CER) that rely on crowdsourcing; we also describe our own experiences of using Amazon MTurk for a CER study. We discuss challenges in recruiting workers with specific backgrounds—such as no programming experience—and considerations in filtering out unreliable research participants. Combining recommendations from the literature with the lessons that we learned whilst conducting our study, we synthesize advice for researchers in CER who are considering crowdsourcing. In our case study, we did not find widespread foul play by crowdsourced workers and, overall, our experiences and the literature suggest that crowdsourced CER is feasible. It is, however, uncertain to what extent crowdsourced data can produce answers that apply to specific educational contexts. More research is needed before definitive conclusions can be drawn about the validity and generalizability of crowdsourced CER.
Original languageEnglish
Title of host publicationKoli Calling '20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research
PublisherACM
Pages1-5
ISBN (Electronic)978-1-4503-8921-1
DOIs
Publication statusPublished - 19 Nov 2020
MoE publication typeA4 Article in a conference publication
EventKoli Calling - International Conference on Computing Education Research - Virtual, Online
Duration: 19 Nov 202022 Nov 2020
https://www.kolicalling.fi/

Conference

ConferenceKoli Calling - International Conference on Computing Education Research
CityVirtual, Online
Period19/11/202022/11/2020
Internet address

Keywords

  • Crowdsourcing
  • CER

Fingerprint

Dive into the research topics of 'Crowdsourcing in Computing Education Research: Case Amazon MTurk'. Together they form a unique fingerprint.

Cite this