Describing UI Screenshots in Natural Language

Luis A. Leiva; Asutosh Hota; Antti Oulasvirta

doi:10.1145/3564702

Describing UI Screenshots in Natural Language

Luis A. Leiva, Asutosh Hota, Antti Oulasvirta

University of Luxembourg

Research output: Contribution to journal › Article › Scientific › peer-review

3 Citations (Scopus)

149 Downloads (Pure)

Abstract

Being able to describe any user interface (UI) screenshot in natural language can promote understanding of the main purpose of the UI, yet currently it cannot be accomplished with state-of-the-art captioning systems. We introduce XUI, a novel method inspired by the global precedence effect to create informative descriptions of UIs, starting with an overview and then providing fine-grained descriptions about the most salient elements. XUI builds upon computational models for topic classification, visual saliency prediction, and natural language generation (NLG). XUI provides descriptions with up to three different granularity levels that, together, describe what is in the interface and what the user can do with it. We found that XUI descriptions are highly readable, are perceived to accurately describe the UI, and score similarly to human-generated UI descriptions. XUI is available as open-source software.

Original language	English
Article number	19
Number of pages	28
Journal	ACM Transactions on Intelligent Systems and Technology
Volume	14
Issue number	1
DOIs	https://doi.org/10.1145/3564702
Publication status	Published - 9 Nov 2022
MoE publication type	A1 Journal article-refereed

Keywords

Captioning
deep learning
natural language processing
visual saliency

Access to Document

10.1145/3564702

XUIAccepted author manuscript, 7.66 MB

Cite this

@article{092efd3ae7c740299aff850168c9d67d,

title = "Describing UI Screenshots in Natural Language",

abstract = "Being able to describe any user interface (UI) screenshot in natural language can promote understanding of the main purpose of the UI, yet currently it cannot be accomplished with state-of-the-art captioning systems. We introduce XUI, a novel method inspired by the global precedence effect to create informative descriptions of UIs, starting with an overview and then providing fine-grained descriptions about the most salient elements. XUI builds upon computational models for topic classification, visual saliency prediction, and natural language generation (NLG). XUI provides descriptions with up to three different granularity levels that, together, describe what is in the interface and what the user can do with it. We found that XUI descriptions are highly readable, are perceived to accurately describe the UI, and score similarly to human-generated UI descriptions. XUI is available as open-source software.",

keywords = "Captioning, deep learning, natural language processing, visual saliency",

author = "Leiva, {Luis A.} and Asutosh Hota and Antti Oulasvirta",

note = "Funding Information: We acknowledge the computational resources provided by the Aalto Science-IT project. We thank Homayun Afrabandpey, Daniel Buschek, Jussi Jokinen, and J{\"o}rg Tiedemann for reviewing an earlier draft of this article. This work has been supported by the Horizon 2020 FET program of the European Union through the ERA-NET Cofund funding (grant CHIST-ERA-20-BCI-001), the European Innovation Council Pathfinder program (SYMBIOTIK project), and the Academy of Finland (grants 291556, 318559, 310947). Publisher Copyright: {\textcopyright} 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.",

year = "2022",

month = nov,

day = "9",

doi = "10.1145/3564702",

language = "English",

volume = "14",

journal = "ACM Transactions on Intelligent Systems and Technology",

issn = "2157-6904",

publisher = "ACM",

number = "1",

}

TY - JOUR

T1 - Describing UI Screenshots in Natural Language

AU - Leiva, Luis A.

AU - Hota, Asutosh

AU - Oulasvirta, Antti

N1 - Funding Information: We acknowledge the computational resources provided by the Aalto Science-IT project. We thank Homayun Afrabandpey, Daniel Buschek, Jussi Jokinen, and Jörg Tiedemann for reviewing an earlier draft of this article. This work has been supported by the Horizon 2020 FET program of the European Union through the ERA-NET Cofund funding (grant CHIST-ERA-20-BCI-001), the European Innovation Council Pathfinder program (SYMBIOTIK project), and the Academy of Finland (grants 291556, 318559, 310947). Publisher Copyright: © 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.

PY - 2022/11/9

Y1 - 2022/11/9

N2 - Being able to describe any user interface (UI) screenshot in natural language can promote understanding of the main purpose of the UI, yet currently it cannot be accomplished with state-of-the-art captioning systems. We introduce XUI, a novel method inspired by the global precedence effect to create informative descriptions of UIs, starting with an overview and then providing fine-grained descriptions about the most salient elements. XUI builds upon computational models for topic classification, visual saliency prediction, and natural language generation (NLG). XUI provides descriptions with up to three different granularity levels that, together, describe what is in the interface and what the user can do with it. We found that XUI descriptions are highly readable, are perceived to accurately describe the UI, and score similarly to human-generated UI descriptions. XUI is available as open-source software.

AB - Being able to describe any user interface (UI) screenshot in natural language can promote understanding of the main purpose of the UI, yet currently it cannot be accomplished with state-of-the-art captioning systems. We introduce XUI, a novel method inspired by the global precedence effect to create informative descriptions of UIs, starting with an overview and then providing fine-grained descriptions about the most salient elements. XUI builds upon computational models for topic classification, visual saliency prediction, and natural language generation (NLG). XUI provides descriptions with up to three different granularity levels that, together, describe what is in the interface and what the user can do with it. We found that XUI descriptions are highly readable, are perceived to accurately describe the UI, and score similarly to human-generated UI descriptions. XUI is available as open-source software.

KW - Captioning

KW - deep learning

KW - natural language processing

KW - visual saliency

UR - http://www.scopus.com/inward/record.url?scp=85149650525&partnerID=8YFLogxK

U2 - 10.1145/3564702

DO - 10.1145/3564702

M3 - Article

AN - SCOPUS:85149650525

SN - 2157-6904

VL - 14

JO - ACM Transactions on Intelligent Systems and Technology

JF - ACM Transactions on Intelligent Systems and Technology

IS - 1

M1 - 19

ER -

Describing UI Screenshots in Natural Language

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

-: Bayesian Artefact Design

Computational Modelling of Emotional Appraisal in HCI

User Interface Design for the Ageing Population

Science-IT

Cite this

Describing UI Screenshots in Natural Language

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Projects

-: Bayesian Artefact Design

Computational Modelling of Emotional Appraisal in HCI

User Interface Design for the Ageing Population

Equipment

Science-IT

Cite this