Projects per year
Abstract
Being able to describe any user interface (UI) screenshot in natural language can promote understanding of the main purpose of the UI, yet currently it cannot be accomplished with state-of-the-art captioning systems. We introduce XUI, a novel method inspired by the global precedence effect to create informative descriptions of UIs, starting with an overview and then providing fine-grained descriptions about the most salient elements. XUI builds upon computational models for topic classification, visual saliency prediction, and natural language generation (NLG). XUI provides descriptions with up to three different granularity levels that, together, describe what is in the interface and what the user can do with it. We found that XUI descriptions are highly readable, are perceived to accurately describe the UI, and score similarly to human-generated UI descriptions. XUI is available as open-source software.
Original language | English |
---|---|
Article number | 19 |
Number of pages | 28 |
Journal | ACM Transactions on Intelligent Systems and Technology |
Volume | 14 |
Issue number | 1 |
DOIs | |
Publication status | Published - 9 Nov 2022 |
MoE publication type | A1 Journal article-refereed |
Keywords
- Captioning
- deep learning
- natural language processing
- visual saliency
Fingerprint
Dive into the research topics of 'Describing UI Screenshots in Natural Language'. Together they form a unique fingerprint.Projects
- 3 Finished
-
-: Bayesian Artefact Design
Oulasvirta, A. (Principal investigator), Shin, J. (Project Member), Hegemann, L. (Project Member), Todi, K. (Project Member), Putkonen, A.-M. (Project Member), Halasinamara Chandramouli, S. (Project Member), Hassinen, H. (Project Member), Dayama, N. (Project Member), Leiva, L. (Project Member), Laine, M. (Project Member), Zhu, Y. (Project Member), Liao, Y.-C. (Project Member), Peng, Z. (Project Member) & Nioche, A. (Project Member)
01/09/2018 → 31/08/2023
Project: Academy of Finland: Other research funding
-
Computational Modelling of Emotional Appraisal in HCI
Jokinen, J. (Principal investigator)
01/09/2017 → 31/08/2020
Project: Academy of Finland: Other research funding
-
User Interface Design for the Ageing Population
Oulasvirta, A. (Principal investigator), Jokinen, J. (Project Member), Shiripour, M. (Project Member) & Hegemann, L. (Project Member)
01/09/2015 → 31/08/2018
Project: Academy of Finland: Other research funding