DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation

Yi Hao Peng; Faria Huq; Yue Jiang; Jason Wu; Xin Yue Li; Jeffrey P. Bigham; Amy Pavel

doi:10.1007/978-3-031-72691-0_26

DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation

Yi Hao Peng^*, Faria Huq, Yue Jiang, Jason Wu, Xin Yue Li, Jeffrey P. Bigham, Amy Pavel

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

1 Citation (Scopus)

Abstract

Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such understanding computationally has required manual data collection and annotation, which is time-consuming and labor-intensive. To overcome this challenge, we present a method to generate synthetic, structured visuals with target labels using code generation. Our method allows people to create datasets with built-in labels and train models with a small number of human-annotated examples. We demonstrate performance improvements in three tasks for understanding slides and UIs: recognizing visual elements, describing visual content, and classifying visual content types.

Original language	English
Title of host publication	Computer Vision – ECCV 2024 - 18th European Conference, Proceedings
Editors	Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
Publisher	Springer
Pages	466-485
Number of pages	20
ISBN (Print)	978-3-031-72690-3
DOIs	https://doi.org/10.1007/978-3-031-72691-0_26
Publication status	Published - 2024
MoE publication type	A4 Conference publication
Event	European Conference on Computer Vision - Milano, Italy Duration: 29 Sept 2024 → 4 Oct 2024 Conference number: 18

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	15082 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Computer Vision
Abbreviated title	ECCV
Country/Territory	Italy
City	Milano
Period	29/09/2024 → 04/10/2024

Keywords

Synthetic Data
Transfer Learning
Visual Design

Access to Document

10.1007/978-3-031-72691-0_26

Cite this

Peng, Y. H., Huq, F., Jiang, Y., Wu, J., Li, X. Y., Bigham, J. P., & Pavel, A. (2024). DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (Eds.), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings (pp. 466-485). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15082 LNCS). Springer. https://doi.org/10.1007/978-3-031-72691-0_26

Peng, Yi Hao ; Huq, Faria ; Jiang, Yue et al. / DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation. Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. editor / Aleš Leonardis ; Elisa Ricci ; Stefan Roth ; Olga Russakovsky ; Torsten Sattler ; Gül Varol. Springer, 2024. pp. 466-485 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{9f872994ab3a4faaa78c343f236e4e5c,

title = "DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation",

abstract = "Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such understanding computationally has required manual data collection and annotation, which is time-consuming and labor-intensive. To overcome this challenge, we present a method to generate synthetic, structured visuals with target labels using code generation. Our method allows people to create datasets with built-in labels and train models with a small number of human-annotated examples. We demonstrate performance improvements in three tasks for understanding slides and UIs: recognizing visual elements, describing visual content, and classifying visual content types.",

keywords = "Synthetic Data, Transfer Learning, Visual Design",

author = "Peng, {Yi Hao} and Faria Huq and Yue Jiang and Jason Wu and Li, {Xin Yue} and Bigham, {Jeffrey P.} and Amy Pavel",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; European Conference on Computer Vision , ECCV ; Conference date: 29-09-2024 Through 04-10-2024",

year = "2024",

doi = "10.1007/978-3-031-72691-0_26",

language = "English",

isbn = "978-3-031-72690-3",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "466--485",

editor = "Ale{\v s} Leonardis and Elisa Ricci and Stefan Roth and Olga Russakovsky and Torsten Sattler and G{\"u}l Varol",

booktitle = "Computer Vision – ECCV 2024 - 18th European Conference, Proceedings",

address = "Germany",

}

Peng, YH, Huq, F, Jiang, Y, Wu, J, Li, XY, Bigham, JP & Pavel, A 2024, DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation. in A Leonardis, E Ricci, S Roth, O Russakovsky, T Sattler & G Varol (eds), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15082 LNCS, Springer, pp. 466-485, European Conference on Computer Vision , Milano, Italy, 29/09/2024. https://doi.org/10.1007/978-3-031-72691-0_26

DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation. / Peng, Yi Hao; Huq, Faria; Jiang, Yue et al.
Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. ed. / Aleš Leonardis; Elisa Ricci; Stefan Roth; Olga Russakovsky; Torsten Sattler; Gül Varol. Springer, 2024. p. 466-485 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15082 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation

AU - Peng, Yi Hao

AU - Huq, Faria

AU - Jiang, Yue

AU - Wu, Jason

AU - Li, Xin Yue

AU - Bigham, Jeffrey P.

AU - Pavel, Amy

N1 - Conference code: 18

PY - 2024

Y1 - 2024

N2 - Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such understanding computationally has required manual data collection and annotation, which is time-consuming and labor-intensive. To overcome this challenge, we present a method to generate synthetic, structured visuals with target labels using code generation. Our method allows people to create datasets with built-in labels and train models with a small number of human-annotated examples. We demonstrate performance improvements in three tasks for understanding slides and UIs: recognizing visual elements, describing visual content, and classifying visual content types.

AB - Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such understanding computationally has required manual data collection and annotation, which is time-consuming and labor-intensive. To overcome this challenge, we present a method to generate synthetic, structured visuals with target labels using code generation. Our method allows people to create datasets with built-in labels and train models with a small number of human-annotated examples. We demonstrate performance improvements in three tasks for understanding slides and UIs: recognizing visual elements, describing visual content, and classifying visual content types.

KW - Synthetic Data

KW - Transfer Learning

KW - Visual Design

UR - http://www.scopus.com/inward/record.url?scp=85208591070&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-72691-0_26

DO - 10.1007/978-3-031-72691-0_26

M3 - Conference article in proceedings

AN - SCOPUS:85208591070

SN - 978-3-031-72690-3

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 466

EP - 485

BT - Computer Vision – ECCV 2024 - 18th European Conference, Proceedings

A2 - Leonardis, Aleš

A2 - Ricci, Elisa

A2 - Roth, Stefan

A2 - Russakovsky, Olga

A2 - Sattler, Torsten

A2 - Varol, Gül

PB - Springer

T2 - European Conference on Computer Vision

Y2 - 29 September 2024 through 4 October 2024

ER -

Peng YH, Huq F, Jiang Y, Wu J, Li XY, Bigham JP et al. DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation. In Leonardis A, Ricci E, Roth S, Russakovsky O, Sattler T, Varol G, editors, Computer Vision – ECCV 2024 - 18th European Conference, Proceedings. Springer. 2024. p. 466-485. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-72691-0_26