Conversations with GUIs

Kashyap Todi; Luis A. Leiva; Daniel Buschek; Pin Tian; Antti Oulasvirta

doi:10.1145/3461778.3462124

Conversations with GUIs

Kashyap Todi
, Luis A. Leiva
, Daniel Buschek
, Pin Tian
, Antti Oulasvirta

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

13 Citations (Scopus)

213 Downloads (Pure)

Abstract

Annotated datasets of application GUIs contain a wealth of information that can be used for various purposes, from providing inspiration to designers and implementation details to developers to assisting end-users during daily use. However, users often struggle to formulate their needs in a way that computers can understand reliably. To address this, we study how people may interact with such GUI datasets using natural language. We elicit user needs in a survey (N = 120) with three target groups (designers, developers, end-users), providing insights into which capabilities would be useful and how users formulate queries. We contribute a labelled dataset of 1317 user queries, and demonstrate an application of a conversational assistant that interprets these queries and retrieves information from a large-scale GUI dataset. It can (1) suggest GUI screenshots for design ideation, (2) highlight details about particular GUI features for development, and (3) reveal further insights about applications. Our fndings can inform design and implementation of intelligent systems to interact with GUI datasets intuitively.

Original language	English
Title of host publication	DIS 2021 - Proceedings of the 2021 ACM Designing Interactive Systems Conference
Subtitle of host publication	Nowhere and Everywhere
Publisher	ACM
Pages	1447-1457
Number of pages	11
ISBN (Electronic)	9781450384766
DOIs	https://doi.org/10.1145/3461778.3462124
Publication status	Published - 28 Jun 2021
MoE publication type	A4 Conference publication
Event	ACM Conference on Designing Interactive Systems: Nowhere and Everywhere - Virtual, Online, United States Duration: 28 Jun 2021 → 2 Jul 2021

Conference

Conference	ACM Conference on Designing Interactive Systems
Abbreviated title	DIS
Country/Territory	United States
City	Virtual, Online
Period	28/06/2021 → 02/07/2021

Funding

This work was funded by the Department of Communications and Networking (Comnet), Finnish Centre for Artifcial Intelligence (FCAI), Academy of Finland (grant numbers 291556, 310947), and the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transformation (bidt).

Keywords

Chatbots
Conversational assistants
Dataset
GUI
NLP
NLU

Access to Document

10.1145/3461778.3462124

ELEC_Todi_Conversatios_with_guis_dis2021Final published version, 2.2 MB

Cite this

@inproceedings{5d3f4113b1d64cc88d9e626cc16551ad,

title = "Conversations with GUIs",

abstract = "Annotated datasets of application GUIs contain a wealth of information that can be used for various purposes, from providing inspiration to designers and implementation details to developers to assisting end-users during daily use. However, users often struggle to formulate their needs in a way that computers can understand reliably. To address this, we study how people may interact with such GUI datasets using natural language. We elicit user needs in a survey (N = 120) with three target groups (designers, developers, end-users), providing insights into which capabilities would be useful and how users formulate queries. We contribute a labelled dataset of 1317 user queries, and demonstrate an application of a conversational assistant that interprets these queries and retrieves information from a large-scale GUI dataset. It can (1) suggest GUI screenshots for design ideation, (2) highlight details about particular GUI features for development, and (3) reveal further insights about applications. Our fndings can inform design and implementation of intelligent systems to interact with GUI datasets intuitively.",

keywords = "Chatbots, Conversational assistants, Dataset, GUI, NLP, NLU",

author = "Kashyap Todi and Leiva, \{Luis A.\} and Daniel Buschek and Pin Tian and Antti Oulasvirta",

note = "Funding Information: This work was funded by the Department of Communications and Networking (Comnet), Finnish Centre for Artifcial Intelligence (FCAI), Academy of Finland (grant numbers 291556, 310947), and the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transformation (bidt). Publisher Copyright: {\textcopyright} 2021 ACM.; ACM Conference on Designing Interactive Systems : Nowhere and Everywhere, DIS ; Conference date: 28-06-2021 Through 02-07-2021",

year = "2021",

month = jun,

day = "28",

doi = "10.1145/3461778.3462124",

language = "English",

pages = "1447--1457",

booktitle = "DIS 2021 - Proceedings of the 2021 ACM Designing Interactive Systems Conference",

publisher = "ACM",

address = "United States",

}

TY - GEN

T1 - Conversations with GUIs

AU - Todi, Kashyap

AU - Leiva, Luis A.

AU - Buschek, Daniel

AU - Tian, Pin

AU - Oulasvirta, Antti

N1 - Funding Information: This work was funded by the Department of Communications and Networking (Comnet), Finnish Centre for Artifcial Intelligence (FCAI), Academy of Finland (grant numbers 291556, 310947), and the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transformation (bidt). Publisher Copyright: © 2021 ACM.

PY - 2021/6/28

Y1 - 2021/6/28

N2 - Annotated datasets of application GUIs contain a wealth of information that can be used for various purposes, from providing inspiration to designers and implementation details to developers to assisting end-users during daily use. However, users often struggle to formulate their needs in a way that computers can understand reliably. To address this, we study how people may interact with such GUI datasets using natural language. We elicit user needs in a survey (N = 120) with three target groups (designers, developers, end-users), providing insights into which capabilities would be useful and how users formulate queries. We contribute a labelled dataset of 1317 user queries, and demonstrate an application of a conversational assistant that interprets these queries and retrieves information from a large-scale GUI dataset. It can (1) suggest GUI screenshots for design ideation, (2) highlight details about particular GUI features for development, and (3) reveal further insights about applications. Our fndings can inform design and implementation of intelligent systems to interact with GUI datasets intuitively.

AB - Annotated datasets of application GUIs contain a wealth of information that can be used for various purposes, from providing inspiration to designers and implementation details to developers to assisting end-users during daily use. However, users often struggle to formulate their needs in a way that computers can understand reliably. To address this, we study how people may interact with such GUI datasets using natural language. We elicit user needs in a survey (N = 120) with three target groups (designers, developers, end-users), providing insights into which capabilities would be useful and how users formulate queries. We contribute a labelled dataset of 1317 user queries, and demonstrate an application of a conversational assistant that interprets these queries and retrieves information from a large-scale GUI dataset. It can (1) suggest GUI screenshots for design ideation, (2) highlight details about particular GUI features for development, and (3) reveal further insights about applications. Our fndings can inform design and implementation of intelligent systems to interact with GUI datasets intuitively.

KW - Chatbots

KW - Conversational assistants

KW - Dataset

KW - GUI

KW - NLP

KW - NLU

UR - https://www.scopus.com/pages/publications/85110139962

U2 - 10.1145/3461778.3462124

DO - 10.1145/3461778.3462124

M3 - Conference article in proceedings

AN - SCOPUS:85110139962

SP - 1447

EP - 1457

BT - DIS 2021 - Proceedings of the 2021 ACM Designing Interactive Systems Conference

PB - ACM

T2 - ACM Conference on Designing Interactive Systems

Y2 - 28 June 2021 through 2 July 2021

ER -

Conversations with GUIs

Abstract

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

-: Finnish Center for Artificial Intelligence

Computational Modelling of Emotional Appraisal in HCI

User Interface Design for the Ageing Population

Cite this

Conversations with GUIs

Abstract

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Projects

-: Finnish Center for Artificial Intelligence

Computational Modelling of Emotional Appraisal in HCI

User Interface Design for the Ageing Population

Cite this