Crowd-sourcing the evaluation of creativity in conceptual design: A pilot study

Matthew Green, Carolyn Conner Seepersad, Katja Hölttä-Otto

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

14 Citations (Scopus)

Abstract

Creativity is often considered to be a critical aspect of engineering innovation and successful product design. Many methods have been proposed for enhancing creativity, originality, and innovation. When these methods are tested, the experiment often generates large numbers of concepts that must be evaluated by experts in a time-consuming process. Similarly, the increased use of crowd-sourcing for generating concepts often leads to a plethora of alternatives that must be evaluated. Accordingly, engineering design practitioners and researchers alike often find themselves evaluating large numbers of concepts. In this paper, the feasibility of using nonexperts to evaluate engineering creativity is investigated. Dozens of students at two universities are asked to rate the originality of several different solutions to a design problem, for which validated expert ratings are available. Results indicate that it is possible to extract expert-level ratings from the non-expert student raters by focusing on the student raters with excellent inter-rater agreement amongst themselves and by training the students with example problems prior to the rating exercise. These results suggest that it may be possible to evaluate originality reliably with a large set of novice raters, perhaps with a Mechanical Turk type of approach.

Original languageEnglish
Title of host publication2nd Biennial International Conference on Dynamics for Design; 26th International Conference on Design Theory and Methodology
PublisherAmerican Society of Mechanical Engineers (ASME)
Volume7
ISBN (Electronic)9780791846407
DOIs
Publication statusPublished - 2014
MoE publication typeA4 Article in a conference publication
EventASME International Design Engineering Technical Conferences & Computers and Information in Engineering Conference - Buffalo, United States
Duration: 17 Aug 201420 Aug 2014

Conference

ConferenceASME International Design Engineering Technical Conferences & Computers and Information in Engineering Conference
Abbreviated titleIDETC/CIE
CountryUnited States
CityBuffalo
Period17/08/201420/08/2014

Fingerprint Dive into the research topics of 'Crowd-sourcing the evaluation of creativity in conceptual design: A pilot study'. Together they form a unique fingerprint.

Cite this