AccessFixer: Enhancing GUI Accessibility for Low Vision Users With R-GCN Model

• URL: http://arxiv.org/abs/2502.15142v1
• Date: Fri, 21 Feb 2025 01:52:51 GMT
• Title: AccessFixer: Enhancing GUI Accessibility for Low Vision Users With R-GCN Model
• Authors: Mengxi Zhang, Huaxiao Liu, Chunyang Chen, Guangyong Gao, Han Li, Jian Zhao,
• Abstract summary: We propose a novel approach named AccessFixer to fix accessibility issues in Graphical User Interfaces (GUIs)<n>With AccessFixer, the fixed GUIs would have a consistent color palette, uniform intervals, and adequate size changes achieved through coordinated adjustments to the attributes of related components.<n>We apply AccessFixer to 10 open-source apps by submitting the fixed results with pull requests on GitHub.
• Score: 32.47608503609055
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Graphical User Interface (GUI) plays a critical role in the interaction between users and mobile applications (apps), aiming at facilitating the operation process. However, due to the variety of functions and non-standardized design, GUIs might have many accessibility issues, like the size of components being too small or their intervals being narrow. These issues would hinder the operation of low vision users, preventing them from obtaining information accurately and conveniently. Although several technologies and methods have been proposed to address these issues, they are typically confined to issue identification, leaving the resolution in the hands of developers. Moreover, it can be challenging to ensure that the color, size, and interval of the fixed GUIs are appropriately compared to the original ones. In this work, we propose a novel approach named AccessFixer, which utilizes the Relational-Graph Convolutional Neural Network (R-GCN) to simultaneously fix three kinds of accessibility issues, including small sizes, narrow intervals, and low color contrast in GUIs. With AccessFixer, the fixed GUIs would have a consistent color palette, uniform intervals, and adequate size changes achieved through coordinated adjustments to the attributes of related components. Our experiments demonstrate the effectiveness and usefulness of AccessFixer in fixing GUI accessibility issues. After fixing 30 real-world apps, our approach solves an average of 81.2% of their accessibility issues. Also, we apply AccessFixer to 10 open-source apps by submitting the fixed results with pull requests (PRs) on GitHub. The results demonstrate that developers approve of our submitted fixed GUIs, with 8 PRs being merged or under fixing. A user study examines that low vision users host a positive attitude toward the GUIs fixed by our method.

Related papers

• Don't Confuse! Redrawing GUI Navigation Flow in Mobile Apps for Visually Impaired Users [22.747735521796077]
  It remains unclear if visually impaired users, who rely solely on the screen readers to navigate and access app information, can do so in the correct and reasonable order.<n>Considering these issues, we proposed a method named RGNF (Re-draw GUI Navigation Flow)<n>It aimed to enhance the understandability and coherence of accessing the content of each component within the Graphical User Interface (GUI)
  arXiv  Detail & Related papers  (2025-02-21T01:33:04Z)
• Are your apps accessible? A GCN-based accessibility checker for low vision users [22.747735521796077]
  We propose a novel approach, named ALVIN, which represents the Graphical User Interface as a graph and adopts the Graph Convolutional Neural Networks (GCN) to label inaccessible components.<n>Experiments on 48 apps demonstrate the effectiveness of ALVIN, with precision of 83.5%, recall of 78.9%, and F1-score of 81.2%, outperforming baseline methods.
  arXiv  Detail & Related papers  (2025-02-20T06:04:06Z)
• GUIWatcher: Automatically Detecting GUI Lags by Analyzing Mobile Application Screencasts [9.997570370503617]
  The Graphical User Interface (GUI) plays a central role in mobile applications, directly affecting usability and user satisfaction.<n>Poor GUI performance, such as lag or unresponsiveness, can lead to negative user experience and decreased mobile application (app) ratings.<n>We present GUIWatcher, a framework designed to detect GUI lags by analyzing screencasts recorded during mobile app testing.
  arXiv  Detail & Related papers  (2025-02-06T16:43:51Z)
• Zero-Shot Prompting Approaches for LLM-based Graphical User Interface Generation [53.1000575179389]
  We propose a Retrieval-Augmented GUI Generation (RAGG) approach, integrated with an LLM-based GUI retrieval re-ranking and filtering mechanism.<n>In addition, we adapt Prompt Decomposition (PDGG) and Self-Critique (SCGG) for GUI generation.<n>Our evaluation, which encompasses over 3,000 GUI annotations from over 100 crowd-workers with UI/UX experience, shows that SCGG, in contrast to PDGG and RAGG, can lead to more effective GUI generation.
  arXiv  Detail & Related papers  (2024-12-15T22:17:30Z)
• Falcon-UI: Understanding GUI Before Following User Instructions [57.67308498231232]
  We introduce an instruction-free GUI navigation dataset, termed Insight-UI dataset, to enhance model comprehension of GUI environments.<n> Insight-UI dataset is automatically generated from the Common Crawl corpus, simulating various platforms.<n>We develop the GUI agent model Falcon-UI, which is initially pretrained on Insight-UI dataset and subsequently fine-tuned on Android and Web GUI datasets.
  arXiv  Detail & Related papers  (2024-12-12T15:29:36Z)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction [69.57190742976091]
  We introduce Aguvis, a unified vision-based framework for autonomous GUI agents.<n>Our approach leverages image-based observations, and grounding instructions in natural language to visual elements.<n>To address the limitations of previous work, we integrate explicit planning and reasoning within the model.
  arXiv  Detail & Related papers  (2024-12-05T18:58:26Z)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent [80.50062396585004]
  Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. We develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations. ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding.
  arXiv  Detail & Related papers  (2024-11-26T14:29:47Z)
• GUICourse: From General Vision Language Models to Versatile GUI Agents [75.5150601913659]
  We contribute GUICourse, a suite of datasets to train visual-based GUI agents. First, we introduce the GUIEnv dataset to strengthen the OCR and grounding capabilities of VLMs. Then, we introduce the GUIAct and GUIChat datasets to enrich their knowledge of GUI components and interactions.
  arXiv  Detail & Related papers  (2024-06-17T08:30:55Z)
• Vision-Based Mobile App GUI Testing: A Survey [29.042723121518765]
  Vision-based mobile app GUI testing approaches emerged with the development of computer vision technologies. We provide a comprehensive investigation of the state-of-the-art techniques on 271 papers, among which 92 are vision-based studies.
  arXiv  Detail & Related papers  (2023-10-20T14:04:04Z)
• Automated and Context-Aware Repair of Color-Related Accessibility Issues for Android Apps [28.880881834251227]
  We propose Iris, an automated and context-aware repair method to fix color-related accessibility issues for apps. By leveraging a novel context-aware technique, Iris resolves the optimal colors and a vital phase of attribute-to-repair localization. Our experiments unveiled that Iris can achieve a 91.38% repair success rate with high effectiveness and efficiency.
  arXiv  Detail & Related papers  (2023-08-17T15:03:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.