Learning GUI Completions with User-defined Constraints

Lukas Brückner, Luis A. Leiva, Antti Oulasvirta

Research output: Contribution to journalArticleScientificpeer-review

5 Citations (Scopus)
250 Downloads (Pure)

Abstract

A key objective in the design of graphical user interfaces (GUIs) is to ensure consistency across screens of the same product. However, designing a compliant layout is time-consuming and can distract designers from creative thinking. This paper studies layout recommendation methods that fulfill such consistency requirements using machine learning. Given a desired element type and size, the methods suggest element placements following real-world GUI design processes. Consistency requirements are given implicitly through previous layouts from which patterns are to be learned, comparable to existing screens of a software product. We adopt two recently proposed methods for this task, a Graph Neural Network (GNN) and a Transformer model, and compare them with a custom approach based on sequence alignment and nearest neighbor search (kNN). The methods were tested on handcrafted datasets with explicit layout patterns, as well as large-scale public datasets of diverse mobile design layouts. Our results show that our instance-based learning algorithm outperforms both neural network approaches. Ultimately, this work contributes to establishing smarter design tools for professional designers with explainable algorithms that increase their efficacy.

Original languageEnglish
Article number6
Pages (from-to)1-40
Number of pages40
JournalACM Transactions on Interactive Intelligent Systems
Volume12
Issue number1
DOIs
Publication statusPublished - Mar 2022
MoE publication typeA1 Journal article-refereed

Keywords

  • design
  • layout completion
  • layouts
  • machine learning
  • User interfaces

Fingerprint

Dive into the research topics of 'Learning GUI Completions with User-defined Constraints'. Together they form a unique fingerprint.

Cite this