Crowdsourcing in Computing Education Research: Case Amazon MTurk

Arto Hellas*, Albina Zavgorodniaia, Juha Sorva

*Tämän työn vastaava kirjoittaja

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussaConference contributionScientificvertaisarvioitu

30 Lataukset (Pure)


Crowdsourcing platforms such as Amazon MTurk provide access to a human workforce that can be given tasks to complete online for a fee. In this article, we review studies in computing education research (CER) that rely on crowdsourcing; we also describe our own experiences of using Amazon MTurk for a CER study. We discuss challenges in recruiting workers with specific backgrounds—such as no programming experience—and considerations in filtering out unreliable research participants. Combining recommendations from the literature with the lessons that we learned whilst conducting our study, we synthesize advice for researchers in CER who are considering crowdsourcing. In our case study, we did not find widespread foul play by crowdsourced workers and, overall, our experiences and the literature suggest that crowdsourced CER is feasible. It is, however, uncertain to what extent crowdsourced data can produce answers that apply to specific educational contexts. More research is needed before definitive conclusions can be drawn about the validity and generalizability of crowdsourced CER.
OtsikkoKoli Calling '20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research
ISBN (elektroninen)978-1-4503-8921-1
DOI - pysyväislinkit
TilaJulkaistu - 19 marrask. 2020
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisuussa
TapahtumaKoli Calling - International Conference on Computing Education Research - Virtual, Online
Kesto: 19 marrask. 202022 marrask. 2020


ConferenceKoli Calling - International Conference on Computing Education Research
KaupunkiVirtual, Online


Sukella tutkimusaiheisiin 'Crowdsourcing in Computing Education Research: Case Amazon MTurk'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä