PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration

• URL: http://arxiv.org/abs/2607.16602v2
• Date: Tue, 21 Jul 2026 07:58:44 GMT
• Title: PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration
• Abstract summary: Action-conditioned world models are a key component of embodied AI, serving as scalable policy evaluators.<n>We propose PAVXploreRL, a reinforcement learning framework built on a pretrained latent world model.<n>Experiments show that PAVXploreRL consistently outperforms pretrained baselines.
• Score: 49.59412060345528
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Action-conditioned world models are a key component of embodied AI, serving as scalable policy evaluators that reduce reliance on expensive real-world rollouts. To accurately capture diverse action-induced dynamics, such models should satisfy three key objectives-Physical Plausibility (P), Action Adherence (A), and Visual Fidelity (V), collectively referred to as PAV-while remaining robust to both in-distribution (ID) expert demonstrations and out-of-distribution (OOD) actions. However, existing methods primarily rely on ID action-video pairs and pixel-level reconstruction losses, which do not explicitly optimize PAV objectives and generalize poorly beyond expert data. To address this, we propose PAVXploreRL, a reinforcement learning framework built on a pretrained latent world model that explicitly optimizes PAV objectives through reward-driven training. To improve action generalization, our method jointly leverages ID trajectories and noise-driven OOD action exploration, without paired video supervision. Experiments show that PAVXploreRL consistently outperforms pretrained baselines, achieving a 5.6% average gain across benchmarks and producing higher-quality PAV properties. As a policy evaluator, it also yields more reliable performance estimates and reduces the overestimation bias of prior expert-only world models such as Ctrl-World. Code: https://github.com/Social-AI-Studio/PAVXploreRL

Related papers

• ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Driving [19.081114003415863]
  We propose a unified understanding-and-generation framework to enable meaningful exploration and provide dense supervision.<n>We augment trajectory prediction with future RGB and depth image generation as dense world modeling objectives.<n>We incorporate this exploration signal into a safety-gated reward and optimize the policy.
  arXiv  Detail & Related papers  (2026-04-03T04:14:13Z)
• World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry [82.93104394404781]
  General-purpose world models promise scalable policy evaluation, optimization, and planning.<n>We propose World Action Verifier (WAV), a framework that enables world models to identify their own prediction errors and self-improve.
  arXiv  Detail & Related papers  (2026-04-02T12:48:36Z)
• Action Draft and Verify: A Self-Verifying Framework for Vision-Language-Action Model [31.013109374489442]
  Vision-Language-Action (VLA) models have recently demonstrated strong performance across embodied tasks.<n>We propose Action-Draft-and-Verify (ADV): diffusion action expert drafts multiple candidate action chunks, and the VLM selects one by scoring all candidates in a single forward pass with a perplexity-style metric.<n>Under matched backbones, training data, and action-chunk length, ADV improves success rate by +4.3 points in simulation and +19.7 points in real-world over diffusion-based baseline.
  arXiv  Detail & Related papers  (2026-03-18T09:16:20Z)
• Self-Correcting VLA: Online Action Refinement via Sparse World Imagination [55.982504915794514]
  We propose Self-Correcting VLA (SC-VLA), which achieve self-improvement by intrinsically guiding action refinement through sparse imagination.<n>SC-VLA achieve state-of-the-art performance, yielding the highest task throughput with 16% fewer steps and a 9% higher success rate than the best-performing baselines.
  arXiv  Detail & Related papers  (2026-02-25T06:58:06Z)
• VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model [87.75549463328836]
  The goal of this paper is to improve the performance and reliability of vision-language-action (VLA) models through iterative online interaction.<n>We propose a simple iterative improvement algorithm that uses real-world roll-out data to improve the fidelity of the world model.<n>We achieve a 39.2% absolute success rate improvement over the base policy and 11.6% improvement from training with the generated synthetic rollouts.
  arXiv  Detail & Related papers  (2026-02-12T15:21:47Z)
• WorldCompass: Reinforcement Learning for Long-Horizon World Models [81.03997753254023]
  This work presents World, a novel Reinforcement Learning (RL) framework for interactive video-based world models.<n>We introduce three core innovations tailored to the autoregressive video generation paradigm.<n>We show that World significantly improves interaction accuracy and visual fidelity across various scenarios.
  arXiv  Detail & Related papers  (2026-02-09T18:59:47Z)
• Reshaping Action Error Distributions for Reliable Vision-Language-Action Models [69.38615670891038]
  In robotic manipulation, vision-language-action (VLA) models have emerged as a promising paradigm for learning generalizable and scalable robot policies.<n>We focus on continuous-action VLA models and move beyond conventional MSE-based regression by reshaping action error distributions during training.<n>We evaluate our approaches across standard, few-shot, and noisy settings on multiple representative VLA architectures.
  arXiv  Detail & Related papers  (2026-02-04T05:37:09Z)
• NORA-1.5: A Vision-Language-Action Model Trained using World Model- and Action-based Preference Rewards [41.87267797252411]
  Vision-language-action (VLA) models have recently shown promising performance on a variety of embodied tasks, yet they still fall short in reliability and generalization.<n>We introduce NORA-1.5, a VLA model built from the pre-trained NORA backbone by adding to it a flow-matching-based action expert.<n>To further improve robustness and task success, we develop a set of reward models for post-training VLA policies.
  arXiv  Detail & Related papers  (2025-11-18T16:55:48Z)
• Safe, Efficient, and Robust Reinforcement Learning for Ranking and Diffusion Models [2.231476498067998]
  dissertation investigates how reinforcement learning methods can be designed to be safe, sample-efficient, and robust.<n> Framed through the unifying perspective of contextual-bandit RL, the work addresses two major application domains - ranking and recommendation, and text-to-image diffusion models.
  arXiv  Detail & Related papers  (2025-10-17T08:37:38Z)
• Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition [52.232968183793986]
  General Policy Composition (GPC) is a training-free method that enhances performance by combining the distributional scores of multiple pre-trained policies.<n>GPC consistently improves performance and adaptability across a diverse set of tasks.
  arXiv  Detail & Related papers  (2025-10-01T16:05:53Z)
• Reinforcement Learning with Inverse Rewards for World Model Post-training [29.19830208692156]
  We propose Reinforcement Learning with Inverse Rewards to improve action-following in video world models.<n>RLIR derives verifiable reward signals by recovering input actions from generated videos using an Inverse Dynamics Model.
  arXiv  Detail & Related papers  (2025-09-28T16:27:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.