VideoPanda: Video Panoramic Diffusion with Multi-view Attention

• URL: http://arxiv.org/abs/2504.11389v2
• Date: Thu, 17 Apr 2025 22:59:37 GMT
• Title: VideoPanda: Video Panoramic Diffusion with Multi-view Attention
• Authors: Kevin Xie, Amirmojtaba Sabour, Jiahui Huang, Despoina Paschalidou, Greg Klar, Umar Iqbal, Sanja Fidler, Xiaohui Zeng,
• Abstract summary: High resolution panoramic video content is paramount for immersive experiences in Virtual Reality, but is non-trivial to collect as it requires specialized equipment and intricate camera setups.<n>VideoPanda generates more realistic and coherent 360$circ$ panoramas across all input conditions compared to existing methods.
• Score: 57.87428280844657
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High resolution panoramic video content is paramount for immersive experiences in Virtual Reality, but is non-trivial to collect as it requires specialized equipment and intricate camera setups. In this work, we introduce VideoPanda, a novel approach for synthesizing 360$^\circ$ videos conditioned on text or single-view video data. VideoPanda leverages multi-view attention layers to augment a video diffusion model, enabling it to generate consistent multi-view videos that can be combined into immersive panoramic content. VideoPanda is trained jointly using two conditions: text-only and single-view video, and supports autoregressive generation of long-videos. To overcome the computational burden of multi-view video generation, we randomly subsample the duration and camera views used during training and show that the model is able to gracefully generalize to generating more frames during inference. Extensive evaluations on both real-world and synthetic video datasets demonstrate that VideoPanda generates more realistic and coherent 360$^\circ$ panoramas across all input conditions compared to existing methods. Visit the project website at https://research.nvidia.com/labs/toronto-ai/VideoPanda/ for results.

Related papers

• Beyond the Frame: Generating 360° Panoramic Videos from Perspective Videos [64.10180665546237]
  360deg videos offer a more complete perspective of our surroundings.<n>Existing video models excel at producing standard videos, but their ability to generate full panoramic videos remains elusive.<n>We develop a high-quality data filtering pipeline to curate pairwise training data and improve the quality of 360deg video generation.<n> Experimental results demonstrate that our model can generate realistic and coherent 360deg videos from in-the-wild perspective video.
  arXiv  Detail & Related papers  (2025-04-10T17:51:38Z)
• Reangle-A-Video: 4D Video Generation as Video-to-Video Translation [51.328567400947435]
  We introduce Reangle-A-Video, a unified framework for generating synchronized multi-view videos from a single input video. Our method reframes the multi-view video generation task as video-to-videos translation, leveraging publicly available image and video diffusion priors.
  arXiv  Detail & Related papers  (2025-03-12T08:26:15Z)
• SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints [43.14498014617223]
  We propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation. We introduce a multi-view synchronization module to maintain appearance and geometry consistency across different viewpoints. Our method enables intriguing extensions, such as re-rendering a video from novel viewpoints.
  arXiv  Detail & Related papers  (2024-12-10T18:55:17Z)
• DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion [60.45000652592418]
  We propose a novel text-driven panoramic generation framework, DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. We show that DiffPano can generate consistent, diverse panoramic images with given unseen text descriptions and camera poses.
  arXiv  Detail & Related papers  (2024-10-31T17:57:02Z)
• VidPanos: Generative Panoramic Videos from Casual Panning Videos [73.77443496436749]
  Panoramic image stitching provides a unified, wide-angle view of a scene that extends beyond the camera's field of view. We present a method for synthesizing a panoramic video from a casually-captured panning video. Our system can create video panoramas for a range of in-the-wild scenes including people, vehicles, and flowing water.
  arXiv  Detail & Related papers  (2024-10-17T17:53:24Z)
• Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention [62.2447324481159]
  Cavia is a novel framework for camera-controllable, multi-view video generation. Our framework extends the spatial and temporal attention modules, improving both viewpoint and temporal consistency. Cavia is the first of its kind that allows the user to specify distinct camera motion while obtaining object motion.
  arXiv  Detail & Related papers  (2024-10-14T17:46:32Z)
• PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation [39.269864548255576]
  We present a panoramic video dataset, PanoVOS. The dataset provides 150 videos with high video resolutions and diverse motions. We present a Panoramic Space Consistency Transformer (PSCFormer) which can effectively utilize the semantic boundary information of the previous frame for pixel-level matching with the current frame.
  arXiv  Detail & Related papers  (2023-09-21T17:59:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.