Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

Yunpeng Bai; Aleksi Ikkala; Antti Oulasvirta; Shengdong Zhao; Lucia J Wang; Pengzhi Yang; Peisen Xu

doi:10.1145/3613904.3642540

Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

Yunpeng Bai, Aleksi Ikkala, Antti Oulasvirta, Shengdong Zhao, Lucia J Wang, Pengzhi Yang, Peisen Xu

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

28 Downloads (Pure)

Abstract

Optical Head-Mounted Displays (OHMDs) allow users to read digital content while walking. A better understanding of how users allocate attention between these two tasks is crucial for improving OHMD interfaces. This paper introduces a computational model for simulating users’ attention switches between reading and walking. We model users’ decision to deploy visual attention as a hierarchical reinforcement learning problem, wherein a supervisory controller optimizes attention allocation while considering both reading activity and walking safety. Our model simulates the control of eye movements and locomotion as an adaptation to the given task priority, design of digital content, and walking speed. The model replicates key multitasking behaviors during OHMD reading while walking, including attention switches, changes in reading and walking speeds, and reading resumptions.

Original language	English
Title of host publication	CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
Editors	Florian Floyd Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe Toups Dugas, Irina Shklovski
Publisher	ACM
Number of pages	18
ISBN (Electronic)	979-8-4007-0330-0
DOIs	https://doi.org/10.1145/3613904.3642540
Publication status	Published - 11 May 2024
MoE publication type	A4 Conference publication
Event	ACM SIGCHI Annual Conference on Human Factors in Computing Systems - Honolulu, USA, Honolulu, United States Duration: 11 May 2024 → 16 May 2024 https://chi2024.acm.org/

Conference

Conference	ACM SIGCHI Annual Conference on Human Factors in Computing Systems
Abbreviated title	ACM CHI
Country/Territory	United States
City	Honolulu
Period	11/05/2024 → 16/05/2024
Internet address	https://chi2024.acm.org/

Keywords

computational rationality
multitasking
bounded optimal control
heads-up computing
deep reinforcement learning

Access to Document

10.1145/3613904.3642540Licence: CC BY

3613904.3642540Final published version, 3.12 MBLicence: CC BY

Cite this

Bai, Y., Ikkala, A., Oulasvirta, A., Zhao, S., Wang, L. J., Yang, P., & Xu, P. (2024). Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays. In F. F. Mueller, P. Kyburz, J. R. Williamson, C. Sas, M. L. Wilson, P. Toups Dugas, & I. Shklovski (Eds.), CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems Article 79 ACM. https://doi.org/10.1145/3613904.3642540

Bai, Yunpeng ; Ikkala, Aleksi ; Oulasvirta, Antti et al. / Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays. CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. editor / Florian Floyd Mueller ; Penny Kyburz ; Julie R. Williamson ; Corina Sas ; Max L. Wilson ; Phoebe Toups Dugas ; Irina Shklovski. ACM, 2024.

@inproceedings{d560216890a3419590c4af868da5358c,

title = "Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays",

abstract = "Optical Head-Mounted Displays (OHMDs) allow users to read digital content while walking. A better understanding of how users allocate attention between these two tasks is crucial for improving OHMD interfaces. This paper introduces a computational model for simulating users{\textquoteright} attention switches between reading and walking. We model users{\textquoteright} decision to deploy visual attention as a hierarchical reinforcement learning problem, wherein a supervisory controller optimizes attention allocation while considering both reading activity and walking safety. Our model simulates the control of eye movements and locomotion as an adaptation to the given task priority, design of digital content, and walking speed. The model replicates key multitasking behaviors during OHMD reading while walking, including attention switches, changes in reading and walking speeds, and reading resumptions.",

keywords = "computational rationality, multitasking, bounded optimal control, heads-up computing, deep reinforcement learning",

author = "Yunpeng Bai and Aleksi Ikkala and Antti Oulasvirta and Shengdong Zhao and Wang, {Lucia J} and Pengzhi Yang and Peisen Xu",

year = "2024",

month = may,

day = "11",

doi = "10.1145/3613904.3642540",

language = "English",

editor = "Mueller, {Florian Floyd} and Penny Kyburz and Williamson, {Julie R.} and Corina Sas and Wilson, {Max L.} and {Toups Dugas}, Phoebe and Irina Shklovski",

booktitle = "CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems",

publisher = "ACM",

address = "United States",

note = "ACM SIGCHI Annual Conference on Human Factors in Computing Systems, ACM CHI ; Conference date: 11-05-2024 Through 16-05-2024",

url = "https://chi2024.acm.org/",

}

Bai, Y, Ikkala, A , Oulasvirta, A, Zhao, S, Wang, LJ, Yang, P & Xu, P 2024, Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays. in FF Mueller, P Kyburz, JR Williamson, C Sas, ML Wilson, P Toups Dugas & I Shklovski (eds), CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems., 79, ACM, ACM SIGCHI Annual Conference on Human Factors in Computing Systems, Honolulu, Hawaii, United States, 11/05/2024. https://doi.org/10.1145/3613904.3642540

Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays. / Bai, Yunpeng; Ikkala, Aleksi ; Oulasvirta, Antti et al.
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. ed. / Florian Floyd Mueller; Penny Kyburz; Julie R. Williamson; Corina Sas; Max L. Wilson; Phoebe Toups Dugas; Irina Shklovski. ACM, 2024. 79.

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

AU - Bai, Yunpeng

AU - Ikkala, Aleksi

AU - Oulasvirta, Antti

AU - Zhao, Shengdong

AU - Wang, Lucia J

AU - Yang, Pengzhi

AU - Xu, Peisen

PY - 2024/5/11

Y1 - 2024/5/11

N2 - Optical Head-Mounted Displays (OHMDs) allow users to read digital content while walking. A better understanding of how users allocate attention between these two tasks is crucial for improving OHMD interfaces. This paper introduces a computational model for simulating users’ attention switches between reading and walking. We model users’ decision to deploy visual attention as a hierarchical reinforcement learning problem, wherein a supervisory controller optimizes attention allocation while considering both reading activity and walking safety. Our model simulates the control of eye movements and locomotion as an adaptation to the given task priority, design of digital content, and walking speed. The model replicates key multitasking behaviors during OHMD reading while walking, including attention switches, changes in reading and walking speeds, and reading resumptions.

AB - Optical Head-Mounted Displays (OHMDs) allow users to read digital content while walking. A better understanding of how users allocate attention between these two tasks is crucial for improving OHMD interfaces. This paper introduces a computational model for simulating users’ attention switches between reading and walking. We model users’ decision to deploy visual attention as a hierarchical reinforcement learning problem, wherein a supervisory controller optimizes attention allocation while considering both reading activity and walking safety. Our model simulates the control of eye movements and locomotion as an adaptation to the given task priority, design of digital content, and walking speed. The model replicates key multitasking behaviors during OHMD reading while walking, including attention switches, changes in reading and walking speeds, and reading resumptions.

KW - computational rationality

KW - multitasking

KW - bounded optimal control

KW - heads-up computing

KW - deep reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=85194844992&partnerID=8YFLogxK

U2 - 10.1145/3613904.3642540

DO - 10.1145/3613904.3642540

M3 - Conference article in proceedings

BT - CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

A2 - Mueller, Florian Floyd

A2 - Kyburz, Penny

A2 - Williamson, Julie R.

A2 - Sas, Corina

A2 - Wilson, Max L.

A2 - Toups Dugas, Phoebe

A2 - Shklovski, Irina

PB - ACM

T2 - ACM SIGCHI Annual Conference on Human Factors in Computing Systems

Y2 - 11 May 2024 through 16 May 2024

ER -

Bai Y, Ikkala A , Oulasvirta A, Zhao S, Wang LJ, Yang P et al. Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays. In Mueller FF, Kyburz P, Williamson JR, Sas C, Wilson ML, Toups Dugas P, Shklovski I, editors, CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. ACM. 2024. 79 doi: 10.1145/3613904.3642540

Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Subjective Functions: Subjective Functions

HEALED/Kaski S.: Human-steered next-generation machine learning for reviving drug design (HEALED)

MAMAA /Kaski S.: Maximally Autonomous AI Assistant/Kaski S.

Cite this

Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Projects

Subjective Functions: Subjective Functions

HEALED/Kaski S.: Human-steered next-generation machine learning for reviving drug design (HEALED)

MAMAA /Kaski S.: Maximally Autonomous AI Assistant/Kaski S.

Cite this