Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

1 Citation (Scopus)
11 Downloads (Pure)

Abstract

Present-day graphical user interfaces (GUIs) exhibit diverse arrangements of text, graphics, and interactive elements such as buttons and menus, but representations of GUIs have not kept up. They do not encapsulate both semantic and visuo-spatial relationships among elements. To seize machine learning’s potential for GUIs more efficiently, Graph4GUI exploits graph neural networks to capture individual elements’ properties and their semantic—visuo-spatial constraints in a layout. The learned representation demonstrated its effectiveness in multiple tasks, especially generating designs in a challenging GUI autocompletion task, which involved predicting the positions of remaining unplaced elements in a partially completed GUI. The new model’s suggestions showed alignment and visual appeal superior to the baseline method and received higher subjective ratings for preference. Furthermore, we demonstrate the practical benefits and efficiency advantages designers perceive when utilizing our model as an autocompletion plug-in.
Original languageEnglish
Title of host publicationCHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
EditorsFlorian Floyd Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe Toups Dugas, Irina Shklovski
PublisherACM
Number of pages18
ISBN (Electronic)979-8-4007-0330-0
DOIs
Publication statusPublished - 11 May 2024
MoE publication typeA4 Conference publication
EventACM SIGCHI Annual Conference on Human Factors in Computing Systems - Honolulu, USA, Honolulu, United States
Duration: 11 May 202416 May 2024
https://chi2024.acm.org/

Conference

ConferenceACM SIGCHI Annual Conference on Human Factors in Computing Systems
Abbreviated titleACM CHI
Country/TerritoryUnited States
CityHonolulu
Period11/05/202416/05/2024
Internet address

Keywords

  • User Interface Representation
  • Constraint-based Layout
  • Graphical User Interface
  • Graph Neural Networks

Fingerprint

Dive into the research topics of 'Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces'. Together they form a unique fingerprint.

Cite this