AXNav: Replaying Accessibility Tests from Natural Language

Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, Jeffrey Nichols

Tutkimustuotos: Artikkeli kirjassa/konferenssijulkaisussaConference article in proceedingsScientificvertaisarvioitu

14 Lataukset (Pure)


Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs. However, to our knowledge, no one has yet explored the use of LLMs in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes a manual accessibility test instruction in natural language (e.g., “Search for a show in VoiceOver”) as input and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers, we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10-participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.

OtsikkoCHI 2024 - Proceedings of the 2024 CHI Conference on Human Factors in Computing Sytems
ISBN (elektroninen)9798400703300
DOI - pysyväislinkit
TilaJulkaistu - 11 toukok. 2024
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaACM SIGCHI Annual Conference on Human Factors in Computing Systems - Honolulu, USA, Honolulu, Yhdysvallat
Kesto: 11 toukok. 202416 toukok. 2024


NimiConference on Human Factors in Computing Systems - Proceedings


ConferenceACM SIGCHI Annual Conference on Human Factors in Computing Systems
LyhennettäACM CHI


Sukella tutkimusaiheisiin 'AXNav: Replaying Accessibility Tests from Natural Language'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä