HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy

• URL: http://arxiv.org/abs/2609.09941v1
• Date: Wed, 09 Sep 2026 09:33:25 GMT
• Title: HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy
• Abstract summary: Generalist robot policies have demonstrated strong generalization across robotic manipulation tasks.<n>Recent methods improve Visual-Language-Action (VLA) policies through online reinforcement learning on real robots.<n>World models offer a promising alternative by enabling policy optimization with imagined rollouts.<n>We propose HaWMPO, a closed-loop reinforcement learning pipeline for VLA policy post-training with world models.
• Score: 22.534022126501373
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generalist robot policies have demonstrated strong generalization across robotic manipulation tasks, yet their success rates remain limited in com- plex long-horizon scenarios. Recent methods improve Visual-Language-Action (VLA) policies through online reinforcement learning on real robots, but such training relies on costly physical interactions, suffers from low sample efficiency, and may introduce hardware and safety risks. World models offer a promising alternative by enabling policy optimization with imagined rollouts. However, long-horizon rollouts generated by world models often suffer from prediction hal- lucinations, producing biased state transitions that can mislead policy learning. To address this issue, we propose Hallucination-aware World Model-based Pol- icy Optimization (HaWMPO), a closed-loop reinforcement learning pipeline for VLA policy post-training with world models. Specifically, HaWMPO introduces an action-conditioned hallucination-aware model to estimate the reliability of gen- erated image sequences, and incorporates hallucination scores into group relative policy optimization through a Reward-Soft mechanism, suppressing unreliable ac- tion chunks during training. On the LIBERO benchmark, HaWMPO achieves the best average success rate, with gains of 15.0% over the base model and 2.8% over the strongest baseline; real-world experiments on a G1 robot further validate its effectiveness, raising the average success rate on two manipulation tasks from 67.5% to 80.0%.

Related papers

• Q-Learning With World Models [69.63968749241239]
  We propose a framework that leverages world models to perform test-time search over imagined trajectories on top of Q-learning to select high-value actions.<n>QWM significantly outperforms strong prior state-of-the-art methods on both sample efficiency and performance.
  arXiv  Detail & Related papers  (2026-08-17T22:00:42Z)
• Learning while Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies [23.266003019334438]
  Generalist robot policies increasingly benefit from large-scale pretraining, but offline data alone is insufficient for robust real-world deployment.<n>We present Learning While Deploying (LWD), a fleet-scale offline-to-online reinforcement learning framework for continual post-training of generalist Vision-Language-Action (VLA) policies.
  arXiv  Detail & Related papers  (2026-05-01T05:20:26Z)
• Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training [54.896907620476675]
  We propose a post-training framework that uses a learned world model as a reusable corrective substrate for failure-targeted policy improvement.<n>Hi-WM caches intermediate states and supports rollback and branching, allowing a single failure state to be reused for multiple corrective continuations.<n>We evaluate Hi-WM on three real-world manipulation tasks spanning both rigid and deformable object interaction, and on two policy backbones.
  arXiv  Detail & Related papers  (2026-04-23T14:42:54Z)
• Self-Correcting VLA: Online Action Refinement via Sparse World Imagination [55.982504915794514]
  We propose Self-Correcting VLA (SC-VLA), which achieve self-improvement by intrinsically guiding action refinement through sparse imagination.<n>SC-VLA achieve state-of-the-art performance, yielding the highest task throughput with 16% fewer steps and a 9% higher success rate than the best-performing baselines.
  arXiv  Detail & Related papers  (2026-02-25T06:58:06Z)
• WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL [30.884160045861616]
  We propose WoVR, a reliable world-model-based reinforcement learning framework for post-training VLA policies.<n>It improves rollout stability through a controllable action-conditioned video world model.<n>It also reshapes imagined interaction to reduce effective error depth via Keyframe-evolutiond Rollouts.
  arXiv  Detail & Related papers  (2026-02-15T03:48:20Z)
• VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model [87.75549463328836]
  The goal of this paper is to improve the performance and reliability of vision-language-action (VLA) models through iterative online interaction.<n>We propose a simple iterative improvement algorithm that uses real-world roll-out data to improve the fidelity of the world model.<n>We achieve a 39.2% absolute success rate improvement over the base policy and 11.6% improvement from training with the generated synthetic rollouts.
  arXiv  Detail & Related papers  (2026-02-12T15:21:47Z)
• WMPO: World Model-based Policy Optimization for Vision-Language-Action Models [22.01666177489494]
  Vision-Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation.<n>We introduce World-Model-based Policy Optimization (WMPO), a principled framework for on-policy VLA without interacting with the real environment.
  arXiv  Detail & Related papers  (2025-11-12T17:54:09Z)
• Ctrl-World: A Controllable Generative World Model for Robot Manipulation [53.71061464925014]
  Generalist robot policies can perform a wide range of manipulation skills.<n> evaluating and improving their ability with unfamiliar objects and instructions remains a significant challenge.<n>World models offer a promising, scalable alternative by enabling policies to rollout within imagination space.
  arXiv  Detail & Related papers  (2025-10-11T09:13:10Z)
• Offline Robotic World Model: Learning Robotic Policies without a Physics Simulator [50.191655141020505]
  Reinforcement Learning (RL) has demonstrated impressive capabilities in robotic control but remains challenging due to high sample complexity, safety concerns, and the sim-to-real gap.<n>We introduce Offline Robotic World Model (RWM-O), a model-based approach that explicitly estimates uncertainty to improve policy learning without reliance on a physics simulator.
  arXiv  Detail & Related papers  (2025-04-23T12:58:15Z)
• ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy [10.596344084789434]
  Vision-Language-Action (VLA) models have shown substantial potential in real-world robotic manipulation.<n>Fine-tuning these models through supervised learning struggles to achieve robust performance due to limited, inconsistent demonstrations.<n>We propose a reinforced fine-tuning approach for VLA models, named ConRFT, which consists of offline and online fine-tuning.
  arXiv  Detail & Related papers  (2025-02-08T05:01:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.