VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis

• URL: http://arxiv.org/abs/2604.09330v1
• Date: Fri, 10 Apr 2026 13:59:54 GMT
• Title: VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis
• Abstract summary: We propose a unified flow-matching-based dual-stream framework that jointly generates video and action under visual and language conditioning.<n>Across both simulated and real-world settings, VAG produces aligned video-action pairs with competitive prediction quality, supports executable trajectory replay, and provides useful synthetic pretraining data.
• Score: 35.55805069125473
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in robot foundation models trained on large-scale human teleoperation data have enabled robots to perform increasingly complex real-world tasks. However, scaling these systems remains difficult because collecting task-specific demonstrations is expensive and labor-intensive. Synthetic data, especially generated videos, offer a promising direction, but existing World Models (WMs) are not directly suitable for policy learning since they do not provide paired action trajectories. World-Action (WA) models partially address this by predicting actions with visual outputs, yet often lack strong video-action alignment, while two-stage pipelines that generate video first and then infer actions introduce inefficiency and error accumulation. To address these limitations, we propose VAG, a unified flow-matching-based dual-stream framework that jointly generates video and action under visual and language conditioning. By synchronizing denoising in both branches and using an adaptive 3D pooling mechanism to transfer compact global video context to the action branch, VAG improves cross-modal consistency during generation. Across both simulated and real-world settings, VAG produces aligned video-action pairs with competitive prediction quality, supports executable trajectory replay, and provides useful synthetic pretraining data that improves downstream policy generalization, indicating its potential as a practical world-action model for embodied data synthesis.

Related papers

• Vid2WAM: Distilling Video Diffusion Priors into World Action Models [59.28345153395937]
  World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions.<n>We propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student.<n>We show that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations.
  arXiv  Detail & Related papers  (2026-08-09T08:02:49Z)
• Native Video-Action Pretraining for Generalizable Robot Control [77.59831077077176]
  LingBot-VA 2.0 is a video-action foundation model built from the ground up for embodiment.<n>To meet the demands of high-frequency inference, our model employs a sparse MoE backbone.<n>Real-world deployment validates LingBot-VA 2.0 as a robust foundation model.
  arXiv  Detail & Related papers  (2026-07-09T16:15:43Z)
• UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation [23.675212404889805]
  World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving.<n>Existing cascaded or dual-DiT designs separate video imagination from action prediction.<n>We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Transformer.
  arXiv  Detail & Related papers  (2026-07-06T14:18:18Z)
• GigaWorld-Policy: An Efficient Action-Centered World--Action Model [50.107640832046464]
  GigaWorld-Policy is an action-centered WAM that learns 2D pixel-action dynamics while enabling efficient action decoding, with optional video generation.<n> Experimental results on real-world robotic platforms show that GigaWorld-Policy runs 9x faster than the leading WAM baseline, Motus.
  arXiv  Detail & Related papers  (2026-03-18T00:52:02Z)
• Causal World Modeling for Robot Control [56.31803788587547]
  Video world models provide the ability to imagine the near future by understanding the causality between actions and visual dynamics.<n>We introduce LingBot-VA, an autoregressive diffusion framework that learns frame prediction and policy execution simultaneously.<n>We evaluate our model on both simulation benchmarks and real-world scenarios, where it shows significant promise in long-horizon manipulation, data efficiency in post-training, and strong generalizability to novel configurations.
  arXiv  Detail & Related papers  (2026-01-29T17:07:43Z)
• EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer [64.69014756863331]
  We introduce EchoMotion, a framework designed to model the joint distribution of appearance and human motion.<n>We also propose MVS-RoPE, which offers unified 3D positional encoding for both video and motion tokens.<n>Our findings reveal that explicitly representing human motion is to appearance, significantly boosting the coherence and plausibility of human-centric video generation.
  arXiv  Detail & Related papers  (2025-12-21T17:08:14Z)
• CoVAR: Co-generation of Video and Action for Robotic Manipulation via Multi-Modal Diffusion [27.567059323636112]
  We present a method to generate video-action pairs that follow text instructions, starting from an initial image observation and the robot's joint states.<n>Our approach automatically provides action labels for video diffusion models, overcoming the common lack of action annotations and enabling their full use for robotic policy learning.
  arXiv  Detail & Related papers  (2025-12-17T23:16:02Z)
• mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs [5.109732854501585]
  We introduce mimic-video, a novel Video-Action Model (VAM) that pairs a pretrained Internet-scale video model with a flow matching-based action decoder conditioned on its latent representations.<n>Our approach achieves state-of-the-art performance on simulated and real-world robotic manipulation tasks, improving sample efficiency by 10x and convergence speed by 2x compared to traditional VLA architectures.
  arXiv  Detail & Related papers  (2025-12-17T18:47:31Z)
• Precise Action-to-Video Generation Through Visual Action Prompts [62.951609704196485]
  Action-driven video generation faces a precision-generality trade-off.<n>Agent-centric action signals provide precision at the cost of cross-domain transferability.<n>We "render" actions into precise visual prompts as domain-agnostic representations.
  arXiv  Detail & Related papers  (2025-08-18T17:12:28Z)
• Pre-Trained Video Generative Models as World Simulators [59.546627730477454]
  We propose Dynamic World Simulation (DWS) to transform pre-trained video generative models into controllable world simulators.<n>To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module.<n> Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models.
  arXiv  Detail & Related papers  (2025-02-10T14:49:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.