Hierarchical Reinforcement Learning Explains Task Interleaving Behavior

Christoph Gebhardt*, Antti Oulasvirta, Otmar Hilliges

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

13 Citations (Scopus)
107 Downloads (Pure)

Abstract

How do people decide how long to continue in a task, when to switch, and to which other task? It is known that task interleaving adapts situationally, showing sensitivity to changes in expected rewards, costs, and task boundaries. However, the mechanisms that underpin the decision to stay in a task versus switch away are not thoroughly understood. Previous work has explained task interleaving by greedy heuristics and a policy that maximizes the marginal rate of return. However, it is unclear how such a strategy would allow for adaptation to environments that offer multiple tasks with complex switch costs and delayed rewards. Here, we develop a hierarchical model of supervisory control driven by reinforcement learning (RL). The core assumption is that the supervisory level learns to switch using task-specific approximate utility estimates, which are computed on the lower level. We show that a hierarchically optimal value function decomposition can be learned from experience, even in conditions with multiple tasks and arbitrary and uncertain reward and cost structures. The model also reproduces well-known key phenomena of task interleaving, such as the sensitivity to costs of resumption and immediate as well as delayed in-task rewards. In a demanding task interleaving study with 211 human participants and realistic tasks (reading, mathematics, question-answering, recognition), the model yielded better predictions of individual-level data than a flat (non-hierarchical) RL model and an omniscient-myopic baseline. Corroborating emerging evidence from cognitive neuroscience, our results suggest hierarchical RL as a plausible model of supervisory control in task interleaving.

Original languageEnglish
Pages (from-to)284-304
Number of pages21
JournalComputational Brain & Behavior
Volume4
Issue number3
Early online dateNov 2020
DOIs
Publication statusPublished - Sept 2021
MoE publication typeA1 Journal article-refereed

Keywords

  • Bayesian inference
  • Computational modeling
  • Hierarchical reinforcement learning
  • Hierarchical reinforcement learning model for task interleaving
  • Task interleaving

Fingerprint

Dive into the research topics of 'Hierarchical Reinforcement Learning Explains Task Interleaving Behavior'. Together they form a unique fingerprint.

Cite this