WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning

• URL: http://arxiv.org/abs/2607.29613v1
• Date: Fri, 31 Jul 2026 16:48:45 GMT
• Title: WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning
• Abstract summary: We present the World Critic Model (WCM) forReinforcement Learning post-training.<n>WCM predicts future latent state and estimates values, such that the critic's representation is explicitly trained to capture temporal dynamics.<n>WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings.
• Score: 44.97650507638252
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value estimator that predominantly operates on single-frame observations or single-frame VLM backbone latents, which is a fundamental mismatch with the partially observable nature of robot control. A naive approach to incorporate observation history into the critic incurs exponential complexity with high-dimensional visual space, and still fails because pure scalar-return regression provides insufficient supervision for learning cross-temporal dynamics. We identify the root cause as a state approximation problem: without an explicit world modeling objective, the critic's representation cannot capture the temporal structure needed for accurate value estimation. To address this, we propose the World Critic Model (WCM), built on a lightweight LeJEPA architecture; WCM jointly predicts future latent state and estimates values, such that the critic's representation is explicitly trained to capture temporal dynamics rather than merely regress scalar returns. WCM integrates seamlessly into both on-policy and off-policy training pipelines and is compatible with state-of-the-art VLA backbones including Pi0, Pi0.5, and OpenVLA-OFT. Extensive experiments on 149 tasks across four benchmarks demonstrate that WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings, with particularly strong generalization gains. We further validate WCM on seven real-world manipulation tasks using OpenVLA-OFT and Pi0.5 with off-policy RL, confirming stable deployment across diverse settings.

Related papers

• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models [8.38821563990503]
  Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training.<n>We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models.<n> Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO consistently outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline.
  arXiv  Detail & Related papers  (2026-08-07T15:09:51Z)
• Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models [16.942019996884273]
  Reinforcement learning (RL) has become indispensable for pushing Vision-Language-Action Models (VLAs) beyond static imitation learning.<n>In this work, we show that VLA models possess useful internal evaluative capabilities.<n>We introduce T2VLA (Test-time VLA), an architecture-agnostic test-time RL framework that enables VLA models to achieve self-bootstrapping policy improvement.
  arXiv  Detail & Related papers  (2026-06-29T07:31:41Z)
• PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning [80.94559742826083]
  Reinforcement Learning with Verifiable Rewards (RLVR) traditionally relies on a sparse, outcome-based signal.<n>Recent work shows that providing a fine-grained, model-intrinsic signal effectively improves language reasoning training by providing step-level guidance without costly external models.<n>While effective for unimodal text, we find that naively applying this global reward to vision-language (V-L) reasoning is a suboptimal strategy.<n>We propose Perception-Decomposed Confidence Reward (PDCR), a framework that solves this by aligning the reward structure with the task's heterogeneous nature.
  arXiv  Detail & Related papers  (2026-05-13T12:55:18Z)
• LoopVLA: Learning Sufficiency in Recurrent Refinement for Vision-Language-Action Models [13.30873593845724]
  LoopVLA is a recurrent Vision-Language-Action architecture that learns representation refinement, action prediction, and sufficiency estimation.<n>We show that LoopVLA pushes the efficiency-performance frontier of VLA policies, reducing parameters by 45% and improving inference throughput by up to 1.7 times.
  arXiv  Detail & Related papers  (2026-05-11T03:51:22Z)
• MA-VLCM: A Vision Language Critic Model for Value Estimation of Policies in Multi-Agent Team Settings [2.9017174137438815]
  Multi-agent reinforcement learning (MARL) commonly relies on a centralized critic to estimate the value function.<n>Large vision-language-action models (VLAs) trained on internet-scale data exhibit strong multimodal reasoning and zero-shot generalization capabilities.<n>We propose Multi-Agent Vision-Language-Critic Models (MA-VLCM), a framework that replaces the learned centralized critic in MARL with a pretrained vision-language model.
  arXiv  Detail & Related papers  (2026-03-16T15:29:41Z)
• Act, Think or Abstain: Complexity-Aware Adaptive Inference for Vision-Language-Action Models [7.802379200026965]
  We propose an adaptive framework that dynamically routes VLA execution based on the complexity of the perceived state.<n>Our approach transforms the VLA's vision-language backbone into an active detection tool by projecting latent embeddings into an ensemble of parametric and non-parametric estimators.
  arXiv  Detail & Related papers  (2026-03-05T13:14:41Z)
• Self-Correcting VLA: Online Action Refinement via Sparse World Imagination [55.982504915794514]
  We propose Self-Correcting VLA (SC-VLA), which achieve self-improvement by intrinsically guiding action refinement through sparse imagination.<n>SC-VLA achieve state-of-the-art performance, yielding the highest task throughput with 16% fewer steps and a 9% higher success rate than the best-performing baselines.
  arXiv  Detail & Related papers  (2026-02-25T06:58:06Z)
• Reshaping Action Error Distributions for Reliable Vision-Language-Action Models [69.38615670891038]
  In robotic manipulation, vision-language-action (VLA) models have emerged as a promising paradigm for learning generalizable and scalable robot policies.<n>We focus on continuous-action VLA models and move beyond conventional MSE-based regression by reshaping action error distributions during training.<n>We evaluate our approaches across standard, few-shot, and noisy settings on multiple representative VLA architectures.
  arXiv  Detail & Related papers  (2026-02-04T05:37:09Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• COVLM-RL: Critical Object-Oriented Reasoning for Autonomous Driving Using VLM-Guided Reinforcement Learning [55.83415345423854]
  We propose COVLM-RL, a novel end-to-end driving framework that integrates Critical Object-oriented (CO) reasoning with reinforcement learning (RL)<n>Experiments conducted in the CARLA simulator demonstrate that COVLM-RL significantly improves the success rate by 30% in trained driving environments.
  arXiv  Detail & Related papers  (2025-12-10T06:18:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.