SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation

• URL: http://arxiv.org/abs/2606.18610v2
• Date: Tue, 23 Jun 2026 00:20:02 GMT
• Title: SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation
• Abstract summary: SC3-Eval is a self-consistent video generation recipe that adapts a pre-trained video foundation model into an accurate policy evaluator.<n>We demonstrate SC3-Eval rollouts reproduce the failure modes that policies exhibit in real-world rollouts, supporting fine-grained diagnostic comparison.
• Score: 73.2142090645987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating generalist robot manipulation policies in the real world is expensive, slow, and difficult to scale. Action-conditioned video world models offer a scalable alternative by simulating policy rollouts. Autoregressive rollouts accumulate compounding errors, observations across multiple camera views must remain mutually consistent, and the evaluator must generalize to policies whose behaviors lie outside the training distribution. We address these challenges with SC3-Eval, a self-consistent video generation recipe that adapts a pre-trained video foundation model into an accurate policy evaluator by enforcing three complementary forms of consistency. First, forward-inverse dynamics consistency jointly trains the model to predict frames from actions and to recover actions from frames, anchoring generated rollouts to a physically plausible action manifold and counteracting the drift a forward-only model cannot penalize. Second, cross-view consistency trains the model to inpaint each camera view from the other, keeping the multi-camera observation coherent over long rollouts without any explicit memory mechanism. Third, test-time consistency reuses the inverse dynamics mode at inference as a per-action-chunk uncertainty signal that terminates rollouts whose generated frames drift away from the requested actions. We also demonstrate SC3-Eval rollouts reproduce the failure modes that policies exhibit in real-world rollouts, supporting fine-grained diagnostic comparison rather than aggregate ranking alone. Across seven real-world vision-language-action policies, SC3-Eval attains a closed-loop Pearson correlation of $0.929$ and MMRV of $0.119$, outperforming three strong prior video-model-based baselines, and generalizes to new tasks.

Related papers

• Overcoming Statistical Bias in Action-Controllable World Models [18.890337378174678]
  Action-conditioned world models aim to predict how visual environments evolve under an agent's actions.<n>Future frames are often highly predictable from visual inertia and recurring motion patterns alone.<n>We introduce CoCo, a Counterfactual Consistency framework to enhance action controllability.
  arXiv  Detail & Related papers  (2026-08-05T10:10:36Z)
• KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding [58.541098216700796]
  evaluating the physical consistency of embodied world models is a critical open challenge.<n>Existing frameworks almost exclusively rely on Inverse Dynamics Models(IDMs) for action extraction.<n>We present KineBench, an IDM-free closed-loop benchmark for EWMs.
  arXiv  Detail & Related papers  (2026-07-22T08:04:17Z)
• Native Video-Action Pretraining for Generalizable Robot Control [77.59831077077176]
  LingBot-VA 2.0 is a video-action foundation model built from the ground up for embodiment.<n>To meet the demands of high-frequency inference, our model employs a sparse MoE backbone.<n>Real-world deployment validates LingBot-VA 2.0 as a robust foundation model.
  arXiv  Detail & Related papers  (2026-07-09T16:15:43Z)
• $ω$-EVA: Envision, Verify, and Act with Latent Interactive World Models [28.544954130906135]
  Embodied policies typically map current observations directly to actions, leaving candidate-action consequences implicit.<n>We introduce $$-EVA, a latent interactive world model that realizes an Envision-Verify-Act loop for embodied action generation.<n>Its three-stage framework learns action-conditioned latent dynamics, trains a language-conditioned flow policy, and feeds the policy's proposal back through the world model.
  arXiv  Detail & Related papers  (2026-06-08T13:12:56Z)
• VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis [35.55805069125473]
  We propose a unified flow-matching-based dual-stream framework that jointly generates video and action under visual and language conditioning.<n>Across both simulated and real-world settings, VAG produces aligned video-action pairs with competitive prediction quality, supports executable trajectory replay, and provides useful synthetic pretraining data.
  arXiv  Detail & Related papers  (2026-04-10T13:59:54Z)
• Persistent Robot World Models: Stabilizing Multi-Step Rollouts via Reinforcement Learning [18.397872306430006]
  Action-conditioned robot world models generate future video frames of the manipulated scene given a robot action sequence.<n>These models are optimized for short-term prediction and break down when deployed autoregressively.<n>We introduce a reinforcement learning scheme that trains the world model on its own autoregressive rollouts.
  arXiv  Detail & Related papers  (2026-03-26T17:36:08Z)
• Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation [53.750389076941396]
  Diffusion-based visuomotor policies capture multimodal action distributions through iterative denoising, but their high inference latency limits real-time robotic control.<n>We propose Ada3Drift, which learns a training-time drifting field that attracts predicted actions toward expert demonstration modes.<n>Ada3Drift achieves state-of-the-art performance while requiring $10times$ fewer function evaluations than diffusion-based alternatives.
  arXiv  Detail & Related papers  (2026-03-12T14:36:53Z)
• DCDM: Divide-and-Conquer Diffusion Models for Consistency-Preserving Video Generation [77.89090846233906]
  We propose a system-level framework, termed the Divide-and-Conquer Diffusion Model (DCDM)<n>DCDM decomposes video consistency modeling into three dedicated components while sharing a unified video generation backbone.<n>We validate our framework on the test set of the CVM Competition at AAAI'26, and the results demonstrate that the proposed strategies effectively address these challenges.
  arXiv  Detail & Related papers  (2026-02-14T07:02:36Z)
• Causal World Modeling for Robot Control [56.31803788587547]
  Video world models provide the ability to imagine the near future by understanding the causality between actions and visual dynamics.<n>We introduce LingBot-VA, an autoregressive diffusion framework that learns frame prediction and policy execution simultaneously.<n>We evaluate our model on both simulation benchmarks and real-world scenarios, where it shows significant promise in long-horizon manipulation, data efficiency in post-training, and strong generalizability to novel configurations.
  arXiv  Detail & Related papers  (2026-01-29T17:07:43Z)
• WorldGym: World Model as An Environment for Policy Evaluation [41.204900701616914]
  WorldGym is an autoregressive, action-conditioned video generation model which serves as a proxy to real world environments.<n> Policies are evaluated via Monte Carlo rollouts in the world model, with a vision-language model providing rewards.<n>We show that WorldGym is able to preserve relative policy rankings across different policy versions, sizes, and training checkpoints.
  arXiv  Detail & Related papers  (2025-05-31T15:51:56Z)
• Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model [63.336123527432136]
  We introduce Bench2Drive-R, a generative framework that enables reactive closed-loop evaluation.<n>Unlike existing video generative models for autonomous driving, the proposed designs are tailored for interactive simulation.<n>We compare the generation quality of Bench2Drive-R with existing generative models and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-11T06:35:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.