GUIrilla: A Scalable Framework for Automated Desktop UI Exploration

• URL: http://arxiv.org/abs/2510.16051v1
• Date: Thu, 16 Oct 2025 19:03:45 GMT
• Title: GUIrilla: A Scalable Framework for Automated Desktop UI Exploration
• Abstract summary: GUIrilla is an automated framework that explores applications via native accessibility APIs to address the critical data collection challenge in GUI automation.<n>We construct and release GUIrilla-Task, a large-scale dataset of 27,171 functionally grounded tasks across 1,108 applications.<n> tuning LLM-based agents on GUIrilla-Task significantly improves performance on downstream UI tasks, outperforming synthetic baselines on the ScreenSpot Pro benchmark while using 97% less data.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Autonomous agents capable of operating complex graphical user interfaces (GUIs) have the potential to transform desktop automation. While recent advances in large language models (LLMs) have significantly improved UI understanding, navigating full-window, multi-application desktop environments remains a major challenge. Data availability is limited by costly manual annotation, closed-source datasets and surface-level synthetic pipelines. We introduce GUIrilla, an automated scalable framework that systematically explores applications via native accessibility APIs to address the critical data collection challenge in GUI automation. Our framework focuses on macOS - an ecosystem with limited representation in current UI datasets - though many of its components are designed for broader cross-platform applicability. GUIrilla organizes discovered interface elements and crawler actions into hierarchical GUI graphs and employs specialized interaction handlers to achieve comprehensive application coverage. Using the application graphs from GUIrilla crawler, we construct and release GUIrilla-Task, a large-scale dataset of 27,171 functionally grounded tasks across 1,108 macOS applications, each annotated with full-desktop and window-level screenshots, accessibility metadata, and semantic action traces. Empirical results show that tuning LLM-based agents on GUIrilla-Task significantly improves performance on downstream UI tasks, outperforming synthetic baselines on the ScreenSpot Pro benchmark while using 97% less data. We also release macapptree, an open-source library for reproducible collection of structured accessibility metadata, along with the full GUIrilla-Task dataset, the manually verified GUIrilla-Gold benchmark, and the framework code to support open research in desktop autonomy.

Related papers

• JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition [51.15720718264743]
  We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device requiring coordinated interaction across platforms.<n>JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device transformations.<n>We show that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world usage.
  arXiv  Detail & Related papers  (2026-09-09T16:59:59Z)
• Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining [29.391227440359287]
  Video2GUI is a framework that extracts GUI interaction trajectories directly from unlabeled Internet videos.<n>We construct WildGUI, a large-scale dataset containing 12 million interaction trajectories spanning over 1,500 applications and websites.<n>We will release both the WildGUI dataset and the Video2GUI pipeline to support future research of GUI agents.
  arXiv  Detail & Related papers  (2026-05-14T12:14:24Z)
• Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Dragging [21.57463393334841]
  dragging the mouse to select and manipulate textual content represents a prevalent and important usage in practical GUI scenarios.<n>We introduce GUI-Drag, a dataset of 161K text dragging examples synthesized through a scalable pipeline.<n>To support systematic and robust evaluation, we construct ScreenDrag, a benchmark with 5,333 examples spanning three levels of interface context.
  arXiv  Detail & Related papers  (2025-11-07T19:40:09Z)
• GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents [59.107657859025586]
  GUI-360$circ$ is a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs)<n>The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications.<n>The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space.
  arXiv  Detail & Related papers  (2025-11-06T12:19:02Z)
• GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning [11.909652592163896]
  GUI-ReWalk is a multi-stage framework for synthesizing realistic and diverse GUI trajectories.<n>By combining randomness with goal-aware reasoning for structure, GUI-ReWalk produces data that better reflects intent-aware, adaptive nature of human-computer interaction.<n>Results demonstrate that GUI-ReWalk enables superior coverage of diverse interaction flows, higher trajectory entropy, and more realistic user intent.
  arXiv  Detail & Related papers  (2025-09-19T08:09:18Z)
• Mobile-Agent-v3: Fundamental Agents for GUI Automation [59.775510710011325]
  This paper introduces a foundational GUI agent model that achieves state-of-the-art performance among open-source end-to-end models.<n>We propose Mobile-Agent-v3, a general-purpose GUI agent framework that further improves performance to 73.3 on AndroidWorld and 37.7 on OSWorld.
  arXiv  Detail & Related papers  (2025-08-21T00:39:12Z)
• Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation [0.0]
  We introduce Screen2AX, the first framework to automatically create real-time, tree-structured accessibility metadata from a single screenshot.<n>Our method uses vision-language and object detection models to detect, describe, and organize UI elements hierarchically, mirroring Desktop's system-level accessibility structure.<n>We demonstrate that Screen2AX delivers a 2.2x performance improvement over native accessibility representations and surpasses the state-of-the-art Omni V2 system on the ScreenSpot benchmark.
  arXiv  Detail & Related papers  (2025-07-22T15:38:12Z)
• MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment [63.62778707277929]
  MobileGUI-RL is a scalable framework that trains GUI agent in online environment.<n>It synthesizes a curriculum of learnable tasks through self-exploration and filtering.<n>It adapts GRPO to GUI navigation with trajectory-aware advantages and composite rewards.
  arXiv  Detail & Related papers  (2025-07-08T07:07:53Z)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction [69.57190742976091]
  Aguvis is a vision-based framework for autonomous GUI agents.<n>It standardizes cross-platform interactions and incorporates structured reasoning via inner monologue.<n>It achieves state-of-the-art performance across offline and real-world online benchmarks.
  arXiv  Detail & Related papers  (2024-12-05T18:58:26Z)
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents [55.37173845836839]
  OS-Atlas is a foundational GUI action model that excels at GUI grounding and OOD agentic tasks. We are releasing the largest open-source cross-platform GUI grounding corpus to date, which contains over 13 million GUI elements.
  arXiv  Detail & Related papers  (2024-10-30T17:10:19Z)
• GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding [73.9254861755974]
  This paper introduces a new dataset, termed GUI-World, which features meticulously crafted Human-MLLM annotations.<n>We evaluate the capabilities of current state-of-the-art MLLMs, including Image LLMs and Video LLMs, in understanding various types of GUI content.
  arXiv  Detail & Related papers  (2024-06-16T06:56:53Z)
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents [17.43878828389188]
  We propose a novel visual Graphical User Interface (GUI) agent, SeeClick, which only relies on screenshots for task automation. To tackle this challenge, we propose to enhance SeeClick with GUI grounding pre-training and devise a method to automate the curation of GUI grounding data. We have also created ScreenSpot, the first realistic GUI grounding benchmark that encompasses mobile, desktop, and web environments.
  arXiv  Detail & Related papers  (2024-01-17T08:10:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.