VideoWeaver: Multimodal Multi-View Video-to-Video Transfer for Embodied Agents

• URL: http://arxiv.org/abs/2603.25420v1
• Date: Thu, 26 Mar 2026 13:14:13 GMT
• Title: VideoWeaver: Multimodal Multi-View Video-to-Video Transfer for Embodied Agents
• Abstract summary: VideoWeaver is the first multimodal multi-view V2V translation framework.<n>We train views at distinct diffusion timesteps, enabling the model to learn both joint and conditional view distributions.<n>Experiments show superior or similar performance to the state-of-the-art on the single-view translation benchmarks.
• Score: 17.66237759970927
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent progress in video-to-video (V2V) translation has enabled realistic resimulation of embodied AI demonstrations, a capability that allows pretrained robot policies to be transferable to new environments without additional data collection. However, prior works can only operate on a single view at a time, while embodied AI tasks are commonly captured from multiple synchronized cameras to support policy learning. Naively applying single-view models independently to each camera leads to inconsistent appearance across views, and standard transformer architectures do not scale to multi-view settings due to the quadratic cost of cross-view attention. We present VideoWeaver, the first multimodal multi-view V2V translation framework. VideoWeaver is initially trained as a single-view flow-based V2V model. To achieve an extension to the multi-view regime, we propose to ground all views in a shared 4D latent space derived from a feed-forward spatial foundation model, namely, Pi3. This encourages view-consistent appearance even under wide baselines and dynamic camera motion. To scale beyond a fixed number of cameras, we train views at distinct diffusion timesteps, enabling the model to learn both joint and conditional view distributions. This in turn allows autoregressive synthesis of new viewpoints conditioned on existing ones. Experiments show superior or similar performance to the state-of-the-art on the single-view translation benchmarks and, for the first time, physically and stylistically consistent multi-view translations, including challenging egocentric and heterogeneous-camera setups central to world randomization for robot learning.

Related papers

• MetaWorld: Scaling Multi-Agent Video World Model from Single-view Video Data [125.43597497646444]
  MetaWorld is a novel framework that scales multi-agent video world models to open-domain environments directly from single-view videos.<n>It achieves superior cross-view consistency and identity fidelity, establishing a highly scalable, physics-driven paradigm for multi-agent video world modeling.
  arXiv  Detail & Related papers  (2026-06-01T18:20:20Z)
• Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning [57.88781687419521]
  Multiple synchronized camera perspectives are used to generate pseudo-demonstrations from each expert trajectory.<n>Experiments in simulation and real-world manipulation tasks demonstrate significant gains in data efficiency and generalization.<n>Our results suggest that scaling camera views provides a practical and scalable solution for imitation learning.
  arXiv  Detail & Related papers  (2026-04-01T07:00:44Z)
• Plenoptic Video Generation [80.3116444692858]
  We introduce PlenopticDreamer, a framework that synchronizes generative hallucinations to maintain synchronization-temporal memory.<n>The core idea is to train a multi-in-out video-conditioned model in an autoregressive manner.<n>Our training incorporates context-scaling to improve convergence, self-conditioning to hallucinations caused by error accumulation, and a long-video conditioning mechanism to support extended video generation.
  arXiv  Detail & Related papers  (2026-01-08T18:58:32Z)
• Reangle-A-Video: 4D Video Generation as Video-to-Video Translation [55.08100087149101]
  We introduce Reangle-A-Video, a unified framework for generating synchronized multi-view videos from a single input video.<n>Our method reframes the multi-view video generation task as video-to-videos translation, leveraging publicly available image and video diffusion priors.
  arXiv  Detail & Related papers  (2025-03-12T08:26:15Z)
• SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints [43.14498014617223]
  We propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation.<n>We introduce a multi-view synchronization module to maintain appearance and geometry consistency across different viewpoints.<n>Our method enables intriguing extensions, such as re-rendering a video from novel viewpoints.
  arXiv  Detail & Related papers  (2024-12-10T18:55:17Z)
• Vivid-ZOO: Multi-View Video Generation with Diffusion Model [76.96449336578286]
  New challenges lie in the lack of massive captioned multi-view videos and the complexity of modeling such multi-dimensional distribution. We propose a novel diffusion-based pipeline that generates high-quality multi-view videos centered around a dynamic 3D object from text.
  arXiv  Detail & Related papers  (2024-06-12T21:44:04Z)
• Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis [43.02778060969546]
  We propose a controllable monocular dynamic view synthesis pipeline. Our model does not require depth as input, and does not explicitly model 3D scene geometry. We believe our framework can potentially unlock powerful applications in rich dynamic scene understanding, perception for robotics, and interactive 3D video viewing experiences for virtual reality.
  arXiv  Detail & Related papers  (2024-05-23T17:59:52Z)
• InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images [83.37640073416749]
  We present a method for learning to generate flythrough videos of natural scenes starting from a single view. This capability is learned from a collection of single photographs, without requiring camera poses or even multiple views of each scene.
  arXiv  Detail & Related papers  (2022-07-22T15:41:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.