UEyes: Understanding Visual Saliency across User Interface Types

Yue Jiang; Luis A. Leiva; Paul R. B. Houssel; Hamed Rezazadegan Tavakoli; Julia Kylmälä; Antti Oulasvirta

doi:10.1145/3544548.3581096

UEyes: Understanding Visual Saliency across User Interface Types

Yue Jiang, Luis A. Leiva, Paul R. B. Houssel, Hamed Rezazadegan Tavakoli, Julia Kylmälä, Antti Oulasvirta

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

34 Citations (Scopus)

92 Downloads (Pure)

Abstract

While user interfaces (UIs) display elements such as images and text in a grid-based layout, UI types differ significantly in the number of elements and how they are displayed. For example, webpage designs rely heavily on images and text, whereas desktop UIs tend to feature numerous small images. To examine how such differences affect the way users look at UIs, we collected and analyzed a large eye-tracking-based dataset, UEyes (62 participants and 1,980 UI screenshots), covering four major UI types: webpage, desktop UI, mobile UI, and poster. We analyze its differences in biases related to such factors as color, location, and gaze direction. We also compare state-of-the-art predictive models and propose improvements for better capturing typical tendencies across UI types. Both the dataset and the models are publicly available.

Original language	English
Title of host publication	CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher	ACM
ISBN (Electronic)	978-1-4503-9421-5
DOIs	https://doi.org/10.1145/3544548.3581096
Publication status	Published - 19 Apr 2023
MoE publication type	A4 Conference publication
Event	ACM SIGCHI Annual Conference on Human Factors in Computing Systems - Hamburg, Germany Duration: 23 Apr 2023 → 28 Apr 2023 https://chi2023.acm.org/

Conference

Conference	ACM SIGCHI Annual Conference on Human Factors in Computing Systems
Abbreviated title	ACM CHI
Country/Territory	Germany
City	Hamburg
Period	23/04/2023 → 28/04/2023
Internet address	https://chi2023.acm.org/

Funding

This work was supported by Aalto University’s Department of Information and Communications Engineering, the Finnish Center for Artifcial Intelligence (FCAI), the Academy of Finland through the projects Human Automata (grant 328813) and BAD (grant 318559), the Horizon 2020 FET program of the European Union (grant CHISTERA-20-BCI-001), and the European Innovation Council Pathfnder program (SYMBIOTIK project, grant 101071147). We appreciate Chuhan Jiao’s initial implementation of the baseline methods for saliency prediction and active discussion with Yao (Marc) Wang.

Keywords

Computer Vision
Deep Learning
Eye Tracking
Human Perception and Cognition
Interaction Design

Access to Document

10.1145/3544548.3581096Licence: CC BY

3544548.3581096Final published version, 35 MBLicence: CC BY

Cite this

@inproceedings{b27d4a9b24fc46c9b645271bba7cdd33,

title = "UEyes: Understanding Visual Saliency across User Interface Types",

abstract = "While user interfaces (UIs) display elements such as images and text in a grid-based layout, UI types differ significantly in the number of elements and how they are displayed. For example, webpage designs rely heavily on images and text, whereas desktop UIs tend to feature numerous small images. To examine how such differences affect the way users look at UIs, we collected and analyzed a large eye-tracking-based dataset, UEyes (62 participants and 1,980 UI screenshots), covering four major UI types: webpage, desktop UI, mobile UI, and poster. We analyze its differences in biases related to such factors as color, location, and gaze direction. We also compare state-of-the-art predictive models and propose improvements for better capturing typical tendencies across UI types. Both the dataset and the models are publicly available.",

keywords = "Computer Vision, Deep Learning, Eye Tracking, Human Perception and Cognition, Interaction Design",

author = "Yue Jiang and Leiva, \{Luis A.\} and \{R. B. Houssel\}, Paul and \{Rezazadegan Tavakoli\}, Hamed and Julia Kylm{\"a}l{\"a} and Antti Oulasvirta",

note = "Funding Information: This work was supported by Aalto University{\textquoteright}s Department of Information and Communications Engineering, the Finnish Center for Artifcial Intelligence (FCAI), the Academy of Finland through the projects Human Automata (grant 328813) and BAD (grant 318559), the Horizon 2020 FET program of the European Union (grant CHISTERA-20-BCI-001), and the European Innovation Council Pathfnder program (SYMBIOTIK project, grant 101071147). We appreciate Chuhan Jiao{\textquoteright}s initial implementation of the baseline methods for saliency prediction and active discussion with Yao (Marc) Wang. Publisher Copyright: {\textcopyright} 2023 Owner/Author.; ACM SIGCHI Annual Conference on Human Factors in Computing Systems , ACM CHI ; Conference date: 23-04-2023 Through 28-04-2023",

year = "2023",

month = apr,

day = "19",

doi = "10.1145/3544548.3581096",

language = "English",

booktitle = "CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems",

publisher = "ACM",

address = "United States",

url = "https://chi2023.acm.org/",

}

Jiang, Y, Leiva, LA, R. B. Houssel, P, Rezazadegan Tavakoli, H, Kylmälä, J & Oulasvirta, A 2023, UEyes: Understanding Visual Saliency across User Interface Types. in CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems., 285, ACM, ACM SIGCHI Annual Conference on Human Factors in Computing Systems , Hamburg, Germany, 23/04/2023. https://doi.org/10.1145/3544548.3581096

TY - GEN

T1 - UEyes

T2 - ACM SIGCHI Annual Conference on Human Factors in Computing Systems

AU - Jiang, Yue

AU - Leiva, Luis A.

AU - R. B. Houssel, Paul

AU - Rezazadegan Tavakoli, Hamed

AU - Kylmälä, Julia

AU - Oulasvirta, Antti

N1 - Funding Information: This work was supported by Aalto University’s Department of Information and Communications Engineering, the Finnish Center for Artifcial Intelligence (FCAI), the Academy of Finland through the projects Human Automata (grant 328813) and BAD (grant 318559), the Horizon 2020 FET program of the European Union (grant CHISTERA-20-BCI-001), and the European Innovation Council Pathfnder program (SYMBIOTIK project, grant 101071147). We appreciate Chuhan Jiao’s initial implementation of the baseline methods for saliency prediction and active discussion with Yao (Marc) Wang. Publisher Copyright: © 2023 Owner/Author.

PY - 2023/4/19

Y1 - 2023/4/19

N2 - While user interfaces (UIs) display elements such as images and text in a grid-based layout, UI types differ significantly in the number of elements and how they are displayed. For example, webpage designs rely heavily on images and text, whereas desktop UIs tend to feature numerous small images. To examine how such differences affect the way users look at UIs, we collected and analyzed a large eye-tracking-based dataset, UEyes (62 participants and 1,980 UI screenshots), covering four major UI types: webpage, desktop UI, mobile UI, and poster. We analyze its differences in biases related to such factors as color, location, and gaze direction. We also compare state-of-the-art predictive models and propose improvements for better capturing typical tendencies across UI types. Both the dataset and the models are publicly available.

AB - While user interfaces (UIs) display elements such as images and text in a grid-based layout, UI types differ significantly in the number of elements and how they are displayed. For example, webpage designs rely heavily on images and text, whereas desktop UIs tend to feature numerous small images. To examine how such differences affect the way users look at UIs, we collected and analyzed a large eye-tracking-based dataset, UEyes (62 participants and 1,980 UI screenshots), covering four major UI types: webpage, desktop UI, mobile UI, and poster. We analyze its differences in biases related to such factors as color, location, and gaze direction. We also compare state-of-the-art predictive models and propose improvements for better capturing typical tendencies across UI types. Both the dataset and the models are publicly available.

KW - Computer Vision

KW - Deep Learning

KW - Eye Tracking

KW - Human Perception and Cognition

KW - Interaction Design

UR - https://www.scopus.com/pages/publications/85158105925

U2 - 10.1145/3544548.3581096

DO - 10.1145/3544548.3581096

M3 - Conference article in proceedings

AN - SCOPUS:85158105925

BT - CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

PB - ACM

Y2 - 23 April 2023 through 28 April 2023

ER -

UEyes: Understanding Visual Saliency across User Interface Types

Abstract

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Human Automata: Simulator-based Methods for Collaborative AI

-: Bayesian Artefact Design

Cite this

UEyes: Understanding Visual Saliency across User Interface Types

Abstract

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Projects

Human Automata: Simulator-based Methods for Collaborative AI

-: Bayesian Artefact Design

Cite this