ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
- URL: http://arxiv.org/abs/2604.11784v1
- Date: Mon, 13 Apr 2026 17:52:04 GMT
- Title: ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
- Abstract summary: textbfClawGUI-RL provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices.<n>textbfClawGUI-Eval enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models.<n>textbfClawGUI-Agent brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory.
- Score: 54.04035382782041
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: GUI agents drive applications through their visual interfaces instead of programmatic APIs, interacting with arbitrary software via taps, swipes, and keystrokes, reaching a long tail of applications that CLI-based agents cannot. Yet progress in this area is bottlenecked less by modeling capacity than by the absence of a coherent full-stack infrastructure: online RL training suffers from environment instability and closed pipelines, evaluation protocols drift silently across works, and trained agents rarely reach real users on real devices. We present \textbf{ClawGUI}, an open-source framework addressing these three gaps within a single harness. \textbf{ClawGUI-RL} provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices, integrating GiGPO with a Process Reward Model for dense step-level supervision. \textbf{ClawGUI-Eval} enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models, achieving 95.8\% reproduction against official baselines. \textbf{ClawGUI-Agent} brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory. Trained end to end within this pipeline, \textbf{ClawGUI-2B} achieves 17.1\% Success Rate on MobileWorld GUI-Only, outperforming the same-scale MAI-UI-2B baseline by 6.0\%.
Related papers
- CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents [57.098070497654874]
Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI.<n>We introduce CUA-Universe, a scalable environment-to-hybrid pipeline.
arXiv Detail & Related papers (2026-09-04T17:26:18Z) - GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments [20.28422479068168]
We introduce GUI-CC, a benchmark that evaluates contextual consistency of GUI world models as agent environments.<n>We construct 500 offline trajectory tasks from GUIOdyssey and 200 emulator-verified online tasks across 30 mobile apps.<n>Experiments show that plausible single-step generation does not guarantee reliable environment simulation.
arXiv Detail & Related papers (2026-08-30T05:19:43Z) - AppAgent-Claw: CLI Is All You Need for GUI Automation [9.116481317329493]
AppAgent-Claw is a demonstration-driven system that converts GUI into reliable, reusable skills without runtime inference.<n>It employs a layered localization strategy to handle visual shifts and a validation-coupled execution model to ensure intended on-screen effects.
arXiv Detail & Related papers (2026-04-15T17:02:09Z) - GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training [30.675339113121225]
We present GUI-GENESIS, a framework to automatically synthesize efficient GUI training environments with verifiable rewards.<n>Experiments show that GUI-GENESIS reduces environment latency by 10 times and costs by over $28,000 per epoch compared to training on real applications.
arXiv Detail & Related papers (2026-02-15T10:58:01Z) - UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning [155.51875080423883]
The development of autonomous agents for graphical user interfaces presents major challenges in artificial intelligence.<n>We present UI-TARS-2, a native GUI-centered agent model that addresses these challenges through a systematic training methodology.<n> Empirical evaluation demonstrates that UI-TARS-2 achieves significant improvements over its predecessor UI-TARS-1.5.
arXiv Detail & Related papers (2025-09-02T17:44:45Z) - Mobile-Agent-v3: Fundamental Agents for GUI Automation [59.775510710011325]
This paper introduces a foundational GUI agent model that achieves state-of-the-art performance among open-source end-to-end models.<n>We propose Mobile-Agent-v3, a general-purpose GUI agent framework that further improves performance to 73.3 on AndroidWorld and 37.7 on OSWorld.
arXiv Detail & Related papers (2025-08-21T00:39:12Z) - MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment [63.62778707277929]
MobileGUI-RL is a scalable framework that trains GUI agent in online environment.<n>It synthesizes a curriculum of learnable tasks through self-exploration and filtering.<n>It adapts GRPO to GUI navigation with trajectory-aware advantages and composite rewards.
arXiv Detail & Related papers (2025-07-08T07:07:53Z) - ZeroGUI: Automating Online GUI Learning at Zero Human Cost [75.21128388931945]
We propose ZeroGUI, a scalable, online learning framework for automating GUI Agent training at Zero human cost.<n>Specifically, ZeroGUI integrates (i) VLM-based automatic task generation to produce diverse training goals from the current environment state, (ii) VLM-based automatic reward estimation to assess task success without hand-crafted evaluation functions, and (iii) two-stage online reinforcement learning to continuously interact with and learn from GUI environments.
arXiv Detail & Related papers (2025-05-29T17:59:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.