UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience

• URL: http://arxiv.org/abs/2603.24533v1
• Date: Wed, 25 Mar 2026 17:10:29 GMT
• Title: UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
• Abstract summary: We propose a novel two-stage self-evolving mobile GUI agent.<n>In the first stage, we employ Rejection Fine-Tuning (RFT), which enables the continuous co-evolution of data and models in a fully autonomous loop.<n>The second stage introduces Group Relative Self-Distillation (GRSD), which identifies critical fork points in group rollouts and constructs dense step-level supervision from successful trajectories to correct failed ones.
• Score: 56.50613398808361
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous mobile GUI agents have attracted increasing attention along with the advancement of Multimodal Large Language Models (MLLMs). However, existing methods still suffer from inefficient learning from failed trajectories and ambiguous credit assignment under sparse rewards for long-horizon GUI tasks. To that end, we propose UI-Voyager, a novel two-stage self-evolving mobile GUI agent. In the first stage, we employ Rejection Fine-Tuning (RFT), which enables the continuous co-evolution of data and models in a fully autonomous loop. The second stage introduces Group Relative Self-Distillation (GRSD), which identifies critical fork points in group rollouts and constructs dense step-level supervision from successful trajectories to correct failed ones. Extensive experiments on AndroidWorld show that our 4B model achieves an 81.0% Pass@1 success rate, outperforming numerous recent baselines and exceeding human-level performance. Ablation and case studies further verify the effectiveness of GRSD. Our method represents a significant leap toward efficient, self-evolving, and high-performance mobile GUI automation without expensive manual data annotation.

Related papers

• Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents [80.03824805228719]
  A major challenge in developing computer-use agents is collecting large-scale, high-quality trajectories.<n>We propose a data-centric paradigm that turns failed trajectories into agent improvements.<n>We validate this approach with the state-of-the-art OpenCUA-72B model on the OSWorld benchmark.
  arXiv  Detail & Related papers  (2026-06-30T07:44:37Z)
• LiteGUI: Distilling Compact GUI Agents with Reinforcement Learning [11.35341244051162]
  Current on-device vision-language GUI agents are constrained by limited model capacity.<n>TraditionalSupervised Fine-Tuning for small-scale models often leads to overfitting, catastrophic forgetting and policy rigidity.<n>We propose a novel SFT-free training paradigm that significantly enhances the performance of small-scale models.
  arXiv  Detail & Related papers  (2026-05-08T09:38:29Z)
• MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux [37.49192877577783]
  We present MagicGUI-RMS, a multi-agent reward model system that delivers adaptive trajectory evaluation, corrective feedback, and self-evolving learning capabilities.<n>To support reward learning at scale, we design a structured data construction pipeline that automatically produces balanced and diverse reward datasets.<n>Experiments demonstrate that MagicGUI-RMS yields substantial gains in task accuracy, behavioral robustness.
  arXiv  Detail & Related papers  (2026-01-19T13:50:43Z)
• EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration [16.593979443102754]
  We introduce EchoTrail-GUI, a novel framework designed to mimic human-like experiential learning by equipping agents with a dynamic, accessible memory.<n>First, an agent autonomously interacts with GUI environments to build a curated database of successful task trajectories, validated by a reward model.<n>Second, in the Memory Injection stage, upon receiving a new task, our system efficiently retrieves the most relevant past trajectories to serve as actionable ''memories''<n>Third, during GUI Task Inference, these memories are injected as in-context guidance to inform the agent's reasoning and decision-making process.
  arXiv  Detail & Related papers  (2025-12-22T13:42:18Z)
• Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control [72.43808515668947]
  We introduce Hi-Agent, a trainable hierarchical vision-language agent for mobile control.<n>Hi-Agent features a high-level reasoning model and a low-level action model that are jointly optimized.<n>Hi-Agent achieves a new State-Of-The-Art (SOTA) 87.9% task success rate on the Android-in-the-Wild (AitW) benchmark.
  arXiv  Detail & Related papers  (2025-10-16T07:38:21Z)
• MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents [36.99267272275733]
  We present an online agentic reinforcement learning framework MobileRL to enhance GUI agents in mobile environments.<n>Its core component is the Difficulty-ADAptive GRPO (ADAGRPO) algorithm.<n>We introduce the shortest-path reward adjustment strategy to reshape rewards concerning the task length in multi-turn agentic tasks.
  arXiv  Detail & Related papers  (2025-09-10T13:09:27Z)
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning [155.51875080423883]
  The development of autonomous agents for graphical user interfaces presents major challenges in artificial intelligence.<n>We present UI-TARS-2, a native GUI-centered agent model that addresses these challenges through a systematic training methodology.<n> Empirical evaluation demonstrates that UI-TARS-2 achieves significant improvements over its predecessor UI-TARS-1.5.
  arXiv  Detail & Related papers  (2025-09-02T17:44:45Z)
• Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation [83.92224427735859]
  We introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution.<n>We develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test.<n>Our model offers significant advantages in critic accuracy compared to current MLLMs.
  arXiv  Detail & Related papers  (2025-06-05T04:12:36Z)
• UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents [37.871793585090586]
  We introduce UI-Genie, a self-improving framework addressing two key challenges in GUI agents.<n> verification of trajectory outcome is challenging and high-quality training data are not scalable.<n>We show that UI-Genie achieves state-of-the-art performance across multiple GUI agent benchmarks.
  arXiv  Detail & Related papers  (2025-05-27T17:58:06Z)
• AutoGLM: Autonomous Foundation Agents for GUIs [51.276965515952]
  We present AutoGLM, a new series in the ChatGLM family, designed to serve as foundation agents for autonomous control of digital devices through Graphical User Interfaces (GUIs) We have developed AutoGLM as a practical foundation agent system for real-world GUI interactions. Our evaluations demonstrate AutoGLM's effectiveness across multiple domains.
  arXiv  Detail & Related papers  (2024-10-28T17:05:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.