WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning

• URL: http://arxiv.org/abs/2608.27508v1
• Date: Thu, 27 Aug 2026 07:09:28 GMT
• Title: WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning
• Abstract summary: We propose WM-R1, a reinforcement learning framework that trains mobile GUI agents with world models instead of real environments.<n>World models serve as the source of state transitions during all rollouts, replacing the real Android environment within the training loop.<n> WM-R1 also embeds world models directly into the thinking process, enabling agents to reason about the consequences of candidate actions before committing to the final action.
• Score: 9.419939770187545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: GUI agents trained with reinforcement learning (RL) have showcased strong environment learning capabilities on mobile platforms. However, RL typically demands extensive real-environment interactions, leading to high resource costs and instability, especially in GUI scenarios. To address these, we propose WM-R1, the first reinforcement learning framework that trains mobile GUI agents with world models instead of real environments. Specifically, world models serve as the source of state transitions during all rollouts, replacing the real Android environment within the training loop. WM-R1 also embeds world models directly into the thinking process, enabling agents to reason about the consequences of candidate actions before committing to the final action. Crucially, WM-R1 eliminates the need for real-environment interaction, supports massively parallelized and step-level granularized trajectory generation grounded in world models, and introduces a multi-dimensional rule-based reward that jointly optimizes task success, trajectory efficiency, and world model utilization. For efficient training, we curate a high-quality dataset of 2000 challenging tasks. Experiments on Android mobile benchmarks demonstrate that WM-R1-trained agents significantly outperform GRPO-only baselines and inference-time simulation methods. Code is available at https://github.com/genalyu/WM-R1 .

Related papers

• How Mobile World Model Guides GUI Agents? [51.29718003718467]
  We train world models across four modalities: delta text, full text, diffusion-based images, and renderable code.<n>These models achieve SoTA performance on both MobileWorldBench and Code2WorldBench.<n>For overconfident mobile agents with low action entropy, posterior self-reflection provides limited gains.
  arXiv  Detail & Related papers  (2026-05-11T10:49:31Z)
• Generalization in Online Reinforcement Learning for Mobile Agents [32.98335803990582]
  We introduce textbfAndroidWorld-Generalization, a benchmark for evaluating zero-shot generalization to unseen task instances, templates, and applications.<n>Experiments on AndroidWorld-Generalization show that RL enables a 7B- parameter VLM agent to surpass supervised fine-tuning baselines.<n>To support and fair comparison, we open-source the full RL training system, including the environment, task suite, models, prompt configurations, and the underlying infrastructure.
  arXiv  Detail & Related papers  (2026-03-08T03:08:11Z)
• Scaling Agent Learning via Experience Synthesis [100.42712232390532]
  Reinforcement learning can empower autonomous agents by enabling self-improvement through interaction.<n>But its practical adoption remains challenging due to costly rollouts, limited task diversity, unreliable reward signals, and infrastructure complexity.<n>We introduce DreamGym, the first unified framework designed to synthesize diverse experiences with scalability in mind.
  arXiv  Detail & Related papers  (2025-11-05T18:58:48Z)
• World-in-World: World Models in a Closed-Loop World [123.85805788728128]
  We introduce World-in-World, the first open platform that benchmarks world models in a closed-loop world that mirrors real agent-environment interactions.<n>We curate four closed-loop environments that rigorously evaluate diverse WMs, prioritize task success as the primary metric, and move beyond the common focus on visual quality.<n>Our study uncovers three surprises: (1) visual quality alone does not guarantee task success, controllability matters more; (2) scaling post-training with action-observation data is more effective than upgrading the pretrained video generators; and (3) allocating more inference-time compute allows WMs to substantially improve closed
  arXiv  Detail & Related papers  (2025-10-20T22:09:15Z)
• Watch and Learn: Learning to Use Computers from Online Videos [50.10702690339142]
  Watch & Learn (W&L) is a framework that converts human demonstration videos readily available on the Internet into executable UI trajectories at scale.<n>We develop an inverse dynamics labeling pipeline with task-aware video retrieval, generate over 53k high-quality trajectories from raw web videos.<n>These results highlight web-scale human demonstration videos as a practical and scalable foundation for advancing CUAs towards real-world deployment.
  arXiv  Detail & Related papers  (2025-10-06T10:29:00Z)
• Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning [35.83999932977034]
  We propose a novel R1-style reinforcement fine-tuning paradigm tailored for next-token prediction models to better align agent behavior with human preferences and evaluation metrics.<n>Our approach introduces a metric-oriented policy optimization algorithm to improve distribution alignment and an iterative "SFT-RFT-SFT" training strategy that alternates between Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT)<n>The results on the Open Sim Agents Challenge showcase that SMART-R1 achieves state-of-the-art performance with an overall realism meta score of 0.7858, ranking first on
  arXiv  Detail & Related papers  (2025-09-28T17:36:13Z)
• Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation [65.3648667980258]
  Vision-language model (VLM) based GUI agents show promise for automating complex tasks, but face significant challenges in applying reinforcement learning (RL)<n>We propose DART, a Decoupled Agentic RL Training framework for GUI agents, which coordinates heterogeneous modules in a highly decoupled manner.<n>On the OSWorld benchmark, DART-GUI-7B achieves a 42.13% task success rate, a 14.61% absolute gain over the base model, and 7.34% higher than open-source SOTA.
  arXiv  Detail & Related papers  (2025-09-28T13:19:20Z)
• UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning [78.86567400365392]
  We present Semi-online Reinforcement Learning, a novel paradigm that simulates online RL on offline trajectories.<n>To capture long-term training signals, Semi-online RL introduces discounted future returns into the reward computation.<n>Experiments show that ours Semi-online RL achieves SOTA performance among 7B models across four dynamic benchmarks.
  arXiv  Detail & Related papers  (2025-09-15T03:24:08Z)
• Mobile-R1: Towards Interactive Reinforcement Learning for VLM-Based Mobile Agent via Task-Level Rewards [29.467847768352282]
  We introduce Mobile-R1, which employs interactive multi-turn reinforcement learning with task-level rewards for mobile agents.<n>Our training framework consists of three stages: initial format finetuning, single-step online training via action-level reward, followed by online training via task-level reward based on multi-turn trajectories.<n>This strategy is designed to enhance the exploration and error correction capabilities of Mobile-R1, leading to significant performance improvements.
  arXiv  Detail & Related papers  (2025-06-25T11:34:43Z)
• GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents [29.65579758217919]
  name is the first reinforcement learning framework designed to enhance the capabilities of LVLMs in high-level real-world task scenarios.<n>name achieves superior performance using only 0.02% of the data compared to previous state-of-the-art methods like OS-Atlas.
  arXiv  Detail & Related papers  (2025-04-14T17:45:54Z)
• Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics [50.191655141020505]
  This work advances model-based reinforcement learning by addressing the challenges of long-horizon prediction, error accumulation, and sim-to-real transfer.<n>By providing a scalable and robust framework, the introduced methods pave the way for adaptive and efficient robotic systems in real-world applications.
  arXiv  Detail & Related papers  (2025-01-17T10:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.