Learning to Assist Agents by Observing Them

Antti Keurulainen; Isak Westerlund; Samuel Kaski; Alexander Ilin

doi:10.1007/978-3-030-86380-7_42

Learning to Assist Agents by Observing Them

Antti Keurulainen^*, Isak Westerlund, Samuel Kaski, Alexander Ilin

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

Abstract

The ability of an AI agent to assist other agents, such as humans, is an important and challenging goal, which requires the assisting agent to reason about the behavior and infer the goals of the assisted agent. Training such an ability by using reinforcement learning usually requires large amounts of online training, which is difficult and costly. On the other hand, offline data about the behavior of the assisted agent might be available, but is non-trivial to take advantage of by methods such as offline reinforcement learning. We introduce methods where the capability to create a representation of the behavior is first pre-trained with offline data, after which only a small amount of interaction data is needed to learn an assisting policy. We test the setting in a gridworld where the helper agent has the capability to manipulate the environment of the assisted artificial agents, and introduce three different scenarios where the assistance considerably improves the performance of the assisted agents.

Original language	English
Title of host publication	Artificial Neural Networks and Machine Learning – ICANN 2021 - 30th International Conference on Artificial Neural Networks, Proceedings
Editors	Igor Farkaš, Paolo Masulli, Sebastian Otte, Stefan Wermter
Publisher	Springer
Pages	519-530
Number of pages	12
ISBN (Print)	978-3-030-86379-1
DOIs	https://doi.org/10.1007/978-3-030-86380-7_42
Publication status	Published - 2021
MoE publication type	A4 Conference publication
Event	International Conference on Artificial Neural Networks - Virtual, Online Duration: 14 Sept 2021 → 17 Sept 2021 Conference number: 30

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12894 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	International Conference on Artificial Neural Networks
Abbreviated title	ICANN
City	Virtual, Online
Period	14/09/2021 → 17/09/2021

Keywords

Cooperative AI
Deep reinforcement learning
Helper agent
Meta-learning
Modelling other agents

Access to Document

10.1007/978-3-030-86380-7_42

https://arxiv.org/abs/2110.01311Licence: CC BY

Cite this

Keurulainen, A., Westerlund, I., Kaski, S., & Ilin, A. (2021). Learning to Assist Agents by Observing Them. In I. Farkaš, P. Masulli, S. Otte, & S. Wermter (Eds.), Artificial Neural Networks and Machine Learning – ICANN 2021 - 30th International Conference on Artificial Neural Networks, Proceedings (pp. 519-530). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12894 LNCS). Springer. https://doi.org/10.1007/978-3-030-86380-7_42

Keurulainen, Antti ; Westerlund, Isak ; Kaski, Samuel et al. / Learning to Assist Agents by Observing Them. Artificial Neural Networks and Machine Learning – ICANN 2021 - 30th International Conference on Artificial Neural Networks, Proceedings. editor / Igor Farkaš ; Paolo Masulli ; Sebastian Otte ; Stefan Wermter. Springer, 2021. pp. 519-530 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{3e9e4434004943e28dc4cb1221e5ffed,

title = "Learning to Assist Agents by Observing Them",

abstract = "The ability of an AI agent to assist other agents, such as humans, is an important and challenging goal, which requires the assisting agent to reason about the behavior and infer the goals of the assisted agent. Training such an ability by using reinforcement learning usually requires large amounts of online training, which is difficult and costly. On the other hand, offline data about the behavior of the assisted agent might be available, but is non-trivial to take advantage of by methods such as offline reinforcement learning. We introduce methods where the capability to create a representation of the behavior is first pre-trained with offline data, after which only a small amount of interaction data is needed to learn an assisting policy. We test the setting in a gridworld where the helper agent has the capability to manipulate the environment of the assisted artificial agents, and introduce three different scenarios where the assistance considerably improves the performance of the assisted agents.",

keywords = "Cooperative AI, Deep reinforcement learning, Helper agent, Meta-learning, Modelling other agents",

author = "Antti Keurulainen and Isak Westerlund and Samuel Kaski and Alexander Ilin",

note = "Funding Information: Acknowledgement. This work was supported by the Academy of Finland (Flag-ship programme: Finnish Center for Artificial Intelligence FCAI, and grants 319264, 292334). Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; International Conference on Artificial Neural Networks, ICANN ; Conference date: 14-09-2021 Through 17-09-2021",

year = "2021",

doi = "10.1007/978-3-030-86380-7_42",

language = "English",

isbn = "978-3-030-86379-1",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "519--530",

editor = "Igor Farka{\v s} and Paolo Masulli and Sebastian Otte and Stefan Wermter",

booktitle = "Artificial Neural Networks and Machine Learning – ICANN 2021 - 30th International Conference on Artificial Neural Networks, Proceedings",

address = "Germany",

}

Keurulainen, A, Westerlund, I, Kaski, S & Ilin, A 2021, Learning to Assist Agents by Observing Them. in I Farkaš, P Masulli, S Otte & S Wermter (eds), Artificial Neural Networks and Machine Learning – ICANN 2021 - 30th International Conference on Artificial Neural Networks, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12894 LNCS, Springer, pp. 519-530, International Conference on Artificial Neural Networks, Virtual, Online, 14/09/2021. https://doi.org/10.1007/978-3-030-86380-7_42

Learning to Assist Agents by Observing Them. / Keurulainen, Antti; Westerlund, Isak; Kaski, Samuel et al.
Artificial Neural Networks and Machine Learning – ICANN 2021 - 30th International Conference on Artificial Neural Networks, Proceedings. ed. / Igor Farkaš; Paolo Masulli; Sebastian Otte; Stefan Wermter. Springer, 2021. p. 519-530 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12894 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - Learning to Assist Agents by Observing Them

AU - Keurulainen, Antti

AU - Westerlund, Isak

AU - Kaski, Samuel

AU - Ilin, Alexander

N1 - Conference code: 30

PY - 2021

Y1 - 2021

N2 - The ability of an AI agent to assist other agents, such as humans, is an important and challenging goal, which requires the assisting agent to reason about the behavior and infer the goals of the assisted agent. Training such an ability by using reinforcement learning usually requires large amounts of online training, which is difficult and costly. On the other hand, offline data about the behavior of the assisted agent might be available, but is non-trivial to take advantage of by methods such as offline reinforcement learning. We introduce methods where the capability to create a representation of the behavior is first pre-trained with offline data, after which only a small amount of interaction data is needed to learn an assisting policy. We test the setting in a gridworld where the helper agent has the capability to manipulate the environment of the assisted artificial agents, and introduce three different scenarios where the assistance considerably improves the performance of the assisted agents.

AB - The ability of an AI agent to assist other agents, such as humans, is an important and challenging goal, which requires the assisting agent to reason about the behavior and infer the goals of the assisted agent. Training such an ability by using reinforcement learning usually requires large amounts of online training, which is difficult and costly. On the other hand, offline data about the behavior of the assisted agent might be available, but is non-trivial to take advantage of by methods such as offline reinforcement learning. We introduce methods where the capability to create a representation of the behavior is first pre-trained with offline data, after which only a small amount of interaction data is needed to learn an assisting policy. We test the setting in a gridworld where the helper agent has the capability to manipulate the environment of the assisted artificial agents, and introduce three different scenarios where the assistance considerably improves the performance of the assisted agents.

KW - Cooperative AI

KW - Deep reinforcement learning

KW - Helper agent

KW - Meta-learning

KW - Modelling other agents

UR - http://www.scopus.com/inward/record.url?scp=85115697678&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-86380-7_42

DO - 10.1007/978-3-030-86380-7_42

M3 - Conference article in proceedings

AN - SCOPUS:85115697678

SN - 978-3-030-86379-1

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 519

EP - 530

BT - Artificial Neural Networks and Machine Learning – ICANN 2021 - 30th International Conference on Artificial Neural Networks, Proceedings

A2 - Farkaš, Igor

A2 - Masulli, Paolo

A2 - Otte, Sebastian

A2 - Wermter, Stefan

PB - Springer

T2 - International Conference on Artificial Neural Networks

Y2 - 14 September 2021 through 17 September 2021

ER -

Keurulainen A, Westerlund I, Kaski S , Ilin A. Learning to Assist Agents by Observing Them. In Farkaš I, Masulli P, Otte S, Wermter S, editors, Artificial Neural Networks and Machine Learning – ICANN 2021 - 30th International Conference on Artificial Neural Networks, Proceedings. Springer. 2021. p. 519-530. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-86380-7_42

Learning to Assist Agents by Observing Them

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Interactive machine learning from multiple biodata sources

Interactive machine learning from multiple biodata sources

Cite this

Learning to Assist Agents by Observing Them

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Projects

Interactive machine learning from multiple biodata sources

Interactive machine learning from multiple biodata sources

Cite this