Coding for culture, diversity, gender, and identity: the potential for automation in research

As engineering education research has evolved, its maturity as a research field is reflected in the literature through the issues that are explored and a growing diversity of contributors and points of view. New questions are being raised around: Who becomes an engineer? Why do people become engineers? How do background characteristics like culture or gender (consolidated in this paper with the word ‘peopleness’), and constructs like diversity or identity influence these questions? We define ‘peopleness’ as “awareness and empathy of the range of people who are affected by a product or service as related to their background characteristics.” This paper investigates how peopleness is being explored in engineering education research.

The primary tool of this study is a program created to rank how closely related a paper was to given words (in this particular study, keywords were related to culture, diversity, gender, and identity). The program mathematically represents words as vectors (specifically, the program uses the GLOVE vectors developed by Jeffrey Pennington, Richard Socher, and Christopher D. Manning of Stanford) such that words with similar meanings have vectors that are closer in euclidean distance. The vectors are used in the program to find documents most likely to relate to a specific topic chosen by the researcher by using word vector distances across a document as a measure of the presence of a topic in a document. A paper could be ranked ‘quite relevant’, ‘slightly relevant’, ‘borderline’, or ‘probably not relevant’. This program was run on engineering education articles, design reports, and transcripts of interviews with recently graduated engineering students about innovation in their work. The population for this study includes articles published in three peer-reviewed and highly focused journals on engineering and engineering education journals from 1996 to 2016: International Journal of Engineering Education (IJEE), Journal of Engineering Education (JEE), and Journal of Women and Minorities in Science and Engineering (JWMSE).

To begin this investigation, we considered publications in these journals during the last ten years, that used the words “culture,” “diversity,” “gender,” or “identity” and picked papers we deemed relevant based on frequency and placement of the keywords. Based on this approach, we identified and considered 118 papers in the JEE sample and 104 papers in the IJEE sample. Since JWMSE’s audience is broader, this study focuses only on the articles related specifically to engineering (including engineering courses and introductory science and mathematics courses typically included in required curriculum for engineering students) to make the samples more comparable. This approach yielded 118 papers in the JWMSE sample. Each publication had different proportions of relevant papers for each keyword.

The program was then run on the hand-coded papers. On average, the program found just 48% of the papers hand-coding deemed relevant to be ‘quite relevant’. When the program was broadened to also include ‘slightly relevant’ papers, the agreement with hand-coding increased to on average over 90%. The agreement varied for each keyword. This paper also outlines two further application examples of the program in the field of engineering to illustrate its potential.

This program has much potential for future use. There is room for further exploration of what differentiates papers that are ‘quite relevant’ and ‘slightly relevant.’ There are also questions of how often the program misses a relevant document and how often it mischaracterizes a document as relevant. Its use for identifying different text mediums could also be explored. In terms of peopleness, there is a good amount of work being done by engineering education researchers. Certain dimensions (gender) have been more explored than others, but even within this focus, there is room for different ways of thinking and researching gender’s impact in engineering education. A more automated means, as proposed in this paper, of identifying how writings and publications are including important topics can be a means of tracking research trends over time.