Amortised Design Optimization for Item Response Theory

Antti Keurulainen; Isak Westerlund; Oskar Keurulainen; Andrew Howes

doi:10.1007/978-3-031-36336-8_56

Amortised Design Optimization for Item Response Theory

Antti Keurulainen^*, Isak Westerlund, Oskar Keurulainen, Andrew Howes

^*Corresponding author for this work

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

Abstract

Item Response Theory (IRT) is a well known method for assessing responses from humans in education and psychology. In education, IRT is used to infer student abilities and characteristics of test items from student responses. Interactions with students are expensive, calling for methods that efficiently gather information for inferring student abilities. Methods based on Optimal Experimental Design (OED) are computationally costly, making them inapplicable for interactive applications. In response, we propose incorporating amortised experimental design into IRT. Here, the computational cost is shifted to a precomputing phase by training a Deep Reinforcement Learning (DRL) agent with synthetic data. The agent is trained to select optimally informative test items for the distribution of students, and to conduct amortised inference conditioned on the experiment outcomes. During deployment the agent estimates parameters from data, and suggests the next test item for the student, in close to real-time, by taking into account the history of experiments and outcomes.

Original language	English
Title of host publication	Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings
Editors	Ning Wang, Genaro Rebolledo-Mendez, Vania Dimitrova, Noboru Matsuda, Olga C. Santos
Publisher	Springer
Pages	359-364
Number of pages	6
ISBN (Print)	9783031363351
DOIs	https://doi.org/10.1007/978-3-031-36336-8_56
Publication status	Published - 2023
MoE publication type	A4 Conference publication
Event	International Conference on Artificial Intelligence in Education - Tokyo, Japan Duration: 3 Jul 2023 → 7 Jul 2023 Conference number: 24

Publication series

Name	Communications in Computer and Information Science
Volume	1831 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	International Conference on Artificial Intelligence in Education
Abbreviated title	AIED
Country/Territory	Japan
City	Tokyo
Period	03/07/2023 → 07/07/2023

Keywords

Deep Reinforcement Learning (DRL)
Experimental Design
Item Response Theory (IRT)

Access to Document

10.1007/978-3-031-36336-8_56

https://arxiv.org/abs/2307.09891

OpenUrl availability

Full text

Cite this

Keurulainen, A., Westerlund, I., Keurulainen, O., & Howes, A. (2023). Amortised Design Optimization for Item Response Theory. In N. Wang, G. Rebolledo-Mendez, V. Dimitrova, N. Matsuda, & O. C. Santos (Eds.), Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings (pp. 359-364). (Communications in Computer and Information Science; Vol. 1831 CCIS). Springer. https://doi.org/10.1007/978-3-031-36336-8_56

Keurulainen, Antti ; Westerlund, Isak ; Keurulainen, Oskar et al. / Amortised Design Optimization for Item Response Theory. Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings. editor / Ning Wang ; Genaro Rebolledo-Mendez ; Vania Dimitrova ; Noboru Matsuda ; Olga C. Santos. Springer, 2023. pp. 359-364 (Communications in Computer and Information Science).

@inproceedings{9c6a5daf37fa4df3aea0b13d57d83b30,

title = "Amortised Design Optimization for Item Response Theory",

abstract = "Item Response Theory (IRT) is a well known method for assessing responses from humans in education and psychology. In education, IRT is used to infer student abilities and characteristics of test items from student responses. Interactions with students are expensive, calling for methods that efficiently gather information for inferring student abilities. Methods based on Optimal Experimental Design (OED) are computationally costly, making them inapplicable for interactive applications. In response, we propose incorporating amortised experimental design into IRT. Here, the computational cost is shifted to a precomputing phase by training a Deep Reinforcement Learning (DRL) agent with synthetic data. The agent is trained to select optimally informative test items for the distribution of students, and to conduct amortised inference conditioned on the experiment outcomes. During deployment the agent estimates parameters from data, and suggests the next test item for the student, in close to real-time, by taking into account the history of experiments and outcomes.",

keywords = "Deep Reinforcement Learning (DRL), Experimental Design, Item Response Theory (IRT)",

author = "Antti Keurulainen and Isak Westerlund and Oskar Keurulainen and Andrew Howes",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; International Conference on Artificial Intelligence in Education, AIED ; Conference date: 03-07-2023 Through 07-07-2023",

year = "2023",

doi = "10.1007/978-3-031-36336-8_56",

language = "English",

isbn = "9783031363351",

series = "Communications in Computer and Information Science",

publisher = "Springer",

pages = "359--364",

editor = "Ning Wang and Genaro Rebolledo-Mendez and Vania Dimitrova and Noboru Matsuda and Santos, {Olga C.}",

booktitle = "Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings",

address = "Germany",

}

Keurulainen, A, Westerlund, I, Keurulainen, O & Howes, A 2023, Amortised Design Optimization for Item Response Theory. in N Wang, G Rebolledo-Mendez, V Dimitrova, N Matsuda & OC Santos (eds), Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings. Communications in Computer and Information Science, vol. 1831 CCIS, Springer, pp. 359-364, International Conference on Artificial Intelligence in Education, Tokyo, Japan, 03/07/2023. https://doi.org/10.1007/978-3-031-36336-8_56

Amortised Design Optimization for Item Response Theory. / Keurulainen, Antti; Westerlund, Isak; Keurulainen, Oskar et al.
Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings. ed. / Ning Wang; Genaro Rebolledo-Mendez; Vania Dimitrova; Noboru Matsuda; Olga C. Santos. Springer, 2023. p. 359-364 (Communications in Computer and Information Science; Vol. 1831 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Amortised Design Optimization for Item Response Theory

AU - Keurulainen, Antti

AU - Westerlund, Isak

AU - Keurulainen, Oskar

AU - Howes, Andrew

N1 - Conference code: 24

PY - 2023

Y1 - 2023

N2 - Item Response Theory (IRT) is a well known method for assessing responses from humans in education and psychology. In education, IRT is used to infer student abilities and characteristics of test items from student responses. Interactions with students are expensive, calling for methods that efficiently gather information for inferring student abilities. Methods based on Optimal Experimental Design (OED) are computationally costly, making them inapplicable for interactive applications. In response, we propose incorporating amortised experimental design into IRT. Here, the computational cost is shifted to a precomputing phase by training a Deep Reinforcement Learning (DRL) agent with synthetic data. The agent is trained to select optimally informative test items for the distribution of students, and to conduct amortised inference conditioned on the experiment outcomes. During deployment the agent estimates parameters from data, and suggests the next test item for the student, in close to real-time, by taking into account the history of experiments and outcomes.

AB - Item Response Theory (IRT) is a well known method for assessing responses from humans in education and psychology. In education, IRT is used to infer student abilities and characteristics of test items from student responses. Interactions with students are expensive, calling for methods that efficiently gather information for inferring student abilities. Methods based on Optimal Experimental Design (OED) are computationally costly, making them inapplicable for interactive applications. In response, we propose incorporating amortised experimental design into IRT. Here, the computational cost is shifted to a precomputing phase by training a Deep Reinforcement Learning (DRL) agent with synthetic data. The agent is trained to select optimally informative test items for the distribution of students, and to conduct amortised inference conditioned on the experiment outcomes. During deployment the agent estimates parameters from data, and suggests the next test item for the student, in close to real-time, by taking into account the history of experiments and outcomes.

KW - Deep Reinforcement Learning (DRL)

KW - Experimental Design

KW - Item Response Theory (IRT)

UR - http://www.scopus.com/inward/record.url?scp=85164947758&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-36336-8_56

DO - 10.1007/978-3-031-36336-8_56

M3 - Conference contribution

AN - SCOPUS:85164947758

SN - 9783031363351

T3 - Communications in Computer and Information Science

SP - 359

EP - 364

BT - Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings

A2 - Wang, Ning

A2 - Rebolledo-Mendez, Genaro

A2 - Dimitrova, Vania

A2 - Matsuda, Noboru

A2 - Santos, Olga C.

PB - Springer

T2 - International Conference on Artificial Intelligence in Education

Y2 - 3 July 2023 through 7 July 2023

ER -

Keurulainen A, Westerlund I, Keurulainen O, Howes A. Amortised Design Optimization for Item Response Theory. In Wang N, Rebolledo-Mendez G, Dimitrova V, Matsuda N, Santos OC, editors, Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky - 24th International Conference, AIED 2023, Proceedings. Springer. 2023. p. 359-364. (Communications in Computer and Information Science). doi: 10.1007/978-3-031-36336-8_56