Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation

• URL: http://arxiv.org/abs/2507.16704v1
• Date: Tue, 22 Jul 2025 15:38:12 GMT
• Title: Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation
• Abstract summary: We introduce Screen2AX, the first framework to automatically create real-time, tree-structured accessibility metadata from a single screenshot.<n>Our method uses vision-language and object detection models to detect, describe, and organize UI elements hierarchically, mirroring Desktop's system-level accessibility structure.<n>We demonstrate that Screen2AX delivers a 2.2x performance improvement over native accessibility representations and surpasses the state-of-the-art Omni V2 system on the ScreenSpot benchmark.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Desktop accessibility metadata enables AI agents to interpret screens and supports users who depend on tools like screen readers. Yet, many applications remain largely inaccessible due to incomplete or missing metadata provided by developers - our investigation shows that only 33% of applications on macOS offer full accessibility support. While recent work on structured screen representation has primarily addressed specific challenges, such as UI element detection or captioning, none has attempted to capture the full complexity of desktop interfaces by replicating their entire hierarchical structure. To bridge this gap, we introduce Screen2AX, the first framework to automatically create real-time, tree-structured accessibility metadata from a single screenshot. Our method uses vision-language and object detection models to detect, describe, and organize UI elements hierarchically, mirroring macOS's system-level accessibility structure. To tackle the limited availability of data for macOS desktop applications, we compiled and publicly released three datasets encompassing 112 macOS applications, each annotated for UI element detection, grouping, and hierarchical accessibility metadata alongside corresponding screenshots. Screen2AX accurately infers hierarchy trees, achieving a 77% F1 score in reconstructing a complete accessibility tree. Crucially, these hierarchy trees improve the ability of autonomous agents to interpret and interact with complex desktop interfaces. We introduce Screen2AX-Task, a benchmark specifically designed for evaluating autonomous agent task execution in macOS desktop environments. Using this benchmark, we demonstrate that Screen2AX delivers a 2.2x performance improvement over native accessibility representations and surpasses the state-of-the-art OmniParser V2 system on the ScreenSpot benchmark.

Related papers

• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments [49.98994180610182]
  Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces.<n>This paper identifies a critical reliability gap: state-of-the-art agents face distinct challenges in real-world desktop environments.<n>We introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness.
  arXiv  Detail & Related papers  (2026-05-13T02:48:52Z)
• OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments [5.613789567241434]
  We introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments.<n>The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications.<n>Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals.
  arXiv  Detail & Related papers  (2026-04-03T08:57:06Z)
• OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent [58.07447442040785]
  We introduce OS-Symphony, a holistic framework that comprises an Orchestrator coordinating two key innovations for robust automation.<n>Results demonstrate that OS-Symphony delivers substantial performance gains across varying model scales.
  arXiv  Detail & Related papers  (2026-01-12T17:55:51Z)
• GUIrilla: A Scalable Framework for Automated Desktop UI Exploration [0.0]
  GUIrilla is an automated framework that explores applications via native accessibility APIs to address the critical data collection challenge in GUI automation.<n>We construct and release GUIrilla-Task, a large-scale dataset of 27,171 functionally grounded tasks across 1,108 applications.<n> tuning LLM-based agents on GUIrilla-Task significantly improves performance on downstream UI tasks, outperforming synthetic baselines on the ScreenSpot Pro benchmark while using 97% less data.
  arXiv  Detail & Related papers  (2025-10-16T19:03:45Z)
• What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilities [56.646832992178105]
  We introduce OmniBench, a cross-platform, graph-based benchmark with an automated pipeline for synthesizing tasks of controllable complexity.<n>We present OmniEval, a multidimensional evaluation framework that includes subtask-level evaluation, graph-based metrics, and comprehensive tests across 10 capabilities.<n>Our dataset contains 36k graph-structured tasks across 20 scenarios, achieving a 91% human acceptance rate.
  arXiv  Detail & Related papers  (2025-06-10T15:59:38Z)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis [59.83524388782554]
  Graphical user interface (GUI) grounding remains a critical bottleneck in computer use agent development.<n>We introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types.<n>We synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples.
  arXiv  Detail & Related papers  (2025-05-19T15:09:23Z)
• Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents [57.59830804627066]
  We introduce MONDAY, a large-scale dataset of 313K annotated frames from 20K instructional videos capturing real-world mobile OS navigation.<n>Models that include MONDAY in their pre-training phases demonstrate robust cross-platform generalization capabilities.<n>We present an automated framework that leverages publicly available video content to create comprehensive task datasets.
  arXiv  Detail & Related papers  (2025-05-19T02:39:03Z)
• UFO2: The Desktop AgentOS [60.317812905300336]
  UFO2 is a multiagent AgentOS for Windows desktops that elevates into practical, system-level automation.<n>We evaluate UFO2 across over 20 real-world Windows applications, demonstrating substantial improvements in robustness and execution accuracy over prior CUAs.<n>Our results show that deep OS integration unlocks a scalable path toward reliable, user-aligned desktop automation.
  arXiv  Detail & Related papers  (2025-04-20T13:04:43Z)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction [69.57190742976091]
  Aguvis is a vision-based framework for autonomous GUI agents.<n>It standardizes cross-platform interactions and incorporates structured reasoning via inner monologue.<n>It achieves state-of-the-art performance across offline and real-world online benchmarks.
  arXiv  Detail & Related papers  (2024-12-05T18:58:26Z)
• OmniParser for Pure Vision Based GUI Agent [37.911094082816504]
  Power multimodal models like GPT-4V as a general agent on multiple operating systems are largely underestimated due to the lack of a robust screen parsing technique. textsc Omni significantly improves GPT-4V's performance on ScreenSpot benchmark. textsc Omni with screenshot only outperforms GPT-4V baselines requiring additional information outside of screenshot.
  arXiv  Detail & Related papers  (2024-08-01T00:00:43Z)
• Weak-to-Strong 3D Object Detection with X-Ray Distillation [75.47580744933724]
  We propose a versatile technique that seamlessly integrates into any existing framework for 3D Object Detection. X-Ray Distillation with Object-Complete Frames is suitable for both supervised and semi-supervised settings. Our proposed methods surpass state-of-the-art in semi-supervised learning by 1-1.5 mAP.
  arXiv  Detail & Related papers  (2024-03-31T13:09:06Z)
• Computer User Interface Understanding. A New Dataset and a Learning Framework [2.4473568032515147]
  We introduce the harder task of computer UI understanding. We present a dataset with a set of videos where a user is performing a sequence of actions and each image shows the desktop contents at that time point. We also present a framework that is composed of a synthetic sample generation pipeline to augment the dataset with relevant characteristics.
  arXiv  Detail & Related papers  (2024-03-15T10:26:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.