Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.20607v1
• Date: Sat, 21 Mar 2026 02:44:39 GMT
• Title: Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
• Abstract summary: Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction.<n>We propose VLA-MBPO, a practical framework to tackle these problems in VLA finetuning.<n>Our approach has three key design choices: (i) adapting unified multimodal models for data-efficient world modeling; (ii) an interleaved view decoding mechanism to enforce multi-view consistency; and (iii) chunk-level branched rollout to mitigate error compounding.
• Score: 35.944951371813296
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across large multimodal models and model-based RL, we propose VLA-MBPO, a practical framework to tackle these problems in VLA finetuning. Our approach has three key design choices: (i) adapting unified multimodal models (UMMs) for data-efficient world modeling; (ii) an interleaved view decoding mechanism to enforce multi-view consistency; and (iii) chunk-level branched rollout to mitigate error compounding. Theoretical analysis and experiments across simulation and real-world tasks demonstrate that VLA-MBPO significantly improves policy performance and sample efficiency, underscoring its robustness and scalability for real-world robotic deployment.

Related papers

• WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modeling [53.14409745466386]
  World models aim to capture environment dynamics in ways that support perception, reasoning, and action.<n>We introduce textttWorldBagel, a unified VLAW framework built on BAGEL, a modern multimodal unified model.<n>We show that textttWorldBagel consistently outperforms task-specific alternatives and learns action representations that are more structured and semantically aligned with visual and linguistic context.
  arXiv  Detail & Related papers  (2026-07-03T16:18:20Z)
• Sword: Style-Robust World Models as Simulators via Dynamic Latent Bootstrapping for VLA Policy Post-Training [19.013714390803052]
  One approach treats learned World Models as generative simulators, enabling policy optimization entirely within "imagination"<n>However, when deployed as simulators for specific environments such as the LIBERO benchmark, existing World Models often suffer from poor generalization and long-horizon error accumulation.<n>We propose Sword, a robust World Model framework, to mitigate these problems.
  arXiv  Detail & Related papers  (2026-05-08T05:54:33Z)
• OmniVLA-RL: A Vision-Language-Action Model with Spatial Understanding and Online RL [1.880672844596704]
  Visual-Language-Action (VLA) models represent a paradigm shift in embodied AI.<n>Existing frameworks struggle with imprecise perception, suboptimal multimodal fusion, and instability in reinforcement learning.<n>We propose OmniVLA-RL, a novel architecture that leverages a Mix-of-Transformers (MoT) design to integrate reasoning, spatial, and action experts.
  arXiv  Detail & Related papers  (2026-04-20T01:36:58Z)
• Reshaping Action Error Distributions for Reliable Vision-Language-Action Models [69.38615670891038]
  In robotic manipulation, vision-language-action (VLA) models have emerged as a promising paradigm for learning generalizable and scalable robot policies.<n>We focus on continuous-action VLA models and move beyond conventional MSE-based regression by reshaping action error distributions during training.<n>We evaluate our approaches across standard, few-shot, and noisy settings on multiple representative VLA architectures.
  arXiv  Detail & Related papers  (2026-02-04T05:37:09Z)
• The Law of Multi-Model Collaboration: Scaling Limits of Model Ensembling for Large Language Models [54.51795784459866]
  We propose a theoretical framework of performance scaling for multi-model collaboration.<n>We show that multi-model systems follow a power-law scaling with respect to the total parameter count.<n> ensembles of heterogeneous model families achieve better performance scaling than those formed within a single model family.
  arXiv  Detail & Related papers  (2025-12-29T09:55:12Z)
• WMPO: World Model-based Policy Optimization for Vision-Language-Action Models [22.01666177489494]
  Vision-Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation.<n>We introduce World-Model-based Policy Optimization (WMPO), a principled framework for on-policy VLA without interacting with the real environment.
  arXiv  Detail & Related papers  (2025-11-12T17:54:09Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective [54.77404771454794]
  We develop a flexible and robust world model for Multi-Agent Reinforcement Learning (MARL) using diffusion models.<n>Our method, Diffusion-Inspired Multi-Agent world model (DIMA), achieves state-of-the-art performance across multiple multi-agent control benchmarks.
  arXiv  Detail & Related papers  (2025-05-27T09:11:38Z)
• Improving Vision-Language-Action Model with Online Reinforcement Learning [17.043068379668842]
  Recent studies have successfully integrated large vision-language models into low-level robotic control by supervised fine-tuning.<n>We propose iRe-VLA framework, which iterates between Reinforcement Learning and Supervised Learning to effectively improve VLA models.
  arXiv  Detail & Related papers  (2025-01-28T02:53:48Z)
• Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models [106.35361897941898]
  We propose a novel world model for Multi-Agent RL (MARL) that learns decentralized local dynamics for scalability.<n>We also introduce a Perceiver Transformer as an effective solution to enable centralized representation aggregation.<n>Results on Starcraft Multi-Agent Challenge (SMAC) show that it outperforms strong model-free approaches and existing model-based methods in both sample efficiency and overall performance.
  arXiv  Detail & Related papers  (2024-06-22T12:40:03Z)
• ReCoRe: Regularized Contrastive Representation Learning of World Model [21.29132219042405]
  We present a world model that learns invariant features using contrastive unsupervised learning and an intervention-invariant regularizer. Our method outperforms current state-of-the-art model-based and model-free RL methods and significantly improves on out-of-distribution point navigation tasks evaluated on the iGibson benchmark.
  arXiv  Detail & Related papers  (2023-12-14T15:53:07Z)
• VDFD: Multi-Agent Value Decomposition Framework with Disentangled World Model [10.36125908359289]
  We propose a novel model-based multi-agent reinforcement learning approach named Value Decomposition Framework with Disentangled World Model.<n>Our method achieves high sample efficiency and exhibits superior performance compared to other baselines across a wide range of multi-agent learning tasks.
  arXiv  Detail & Related papers  (2023-09-08T22:12:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.