360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model

• URL: http://arxiv.org/abs/2401.06578v2
• Date: Fri, 10 May 2024 12:11:16 GMT
• Title: 360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model
• Authors: Qian Wang, Weiqi Li, Chong Mou, Xinhua Cheng, Jian Zhang,
• Abstract summary: We propose a pipeline named 360-Degree Video Diffusion model (360DVD) for generating 360-degree panoramic videos. We introduce a lightweight 360-Adapter accompanied by 360 Enhancement Techniques to transform pre-trained T2V models for panorama video generation. We also propose a new panorama dataset named WEB360 consisting of panoramic video-text pairs for training 360DVD.
• Score: 23.708946172342067
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Panorama video recently attracts more interest in both study and application, courtesy of its immersive experience. Due to the expensive cost of capturing 360-degree panoramic videos, generating desirable panorama videos by prompts is urgently required. Lately, the emerging text-to-video (T2V) diffusion methods demonstrate notable effectiveness in standard video generation. However, due to the significant gap in content and motion patterns between panoramic and standard videos, these methods encounter challenges in yielding satisfactory 360-degree panoramic videos. In this paper, we propose a pipeline named 360-Degree Video Diffusion model (360DVD) for generating 360-degree panoramic videos based on the given prompts and motion conditions. Specifically, we introduce a lightweight 360-Adapter accompanied by 360 Enhancement Techniques to transform pre-trained T2V models for panorama video generation. We further propose a new panorama dataset named WEB360 consisting of panoramic video-text pairs for training 360DVD, addressing the absence of captioned panoramic video datasets. Extensive experiments demonstrate the superiority and effectiveness of 360DVD for panorama video generation. Our project page is at https://akaneqwq.github.io/360DVD/.

Related papers

• Imagine360: Immersive 360 Video Generation from Perspective Anchor [79.97844408255897]
  Imagine360 is a perspective-to-$360circ$ video generation framework. It learns fine-grained spherical visual and motion patterns from limited $360circ$ video data. It achieves superior graphics quality and motion coherence among state-of-the-art $360circ$ video generation methods.
  arXiv  Detail & Related papers  (2024-12-04T18:50:08Z)
• DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion [60.45000652592418]
  We propose a novel text-driven panoramic generation framework, DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. We show that DiffPano can generate consistent, diverse panoramic images with given unseen text descriptions and camera poses.
  arXiv  Detail & Related papers  (2024-10-31T17:57:02Z)
• VidPanos: Generative Panoramic Videos from Casual Panning Videos [73.77443496436749]
  Panoramic image stitching provides a unified, wide-angle view of a scene that extends beyond the camera's field of view. We present a method for synthesizing a panoramic video from a casually-captured panning video. Our system can create video panoramas for a range of in-the-wild scenes including people, vehicles, and flowing water.
  arXiv  Detail & Related papers  (2024-10-17T17:53:24Z)
• SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting [53.32467009064287]
  We propose a text-driven 3D-consistent scene generation model: SceneDreamer360. Our proposed method leverages a text-driven panoramic image generation model as a prior for 3D scene generation. Our experiments demonstrate that SceneDreamer360 with its panoramic image generation and 3DGS can produce higher quality, spatially consistent, and visually appealing 3D scenes from any text prompt.
  arXiv  Detail & Related papers  (2024-08-25T02:56:26Z)
• SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix [60.48666051245761]
  We propose a pose-free and training-free approach for generating 3D stereoscopic videos. Our method warps a generated monocular video into camera views on stereoscopic baseline using estimated video depth. We develop a disocclusion boundary re-injection scheme that further improves the quality of video inpainting.
  arXiv  Detail & Related papers  (2024-06-29T08:33:55Z)
• See360: Novel Panoramic View Interpolation [24.965259708297932]
  See360 is a versatile and efficient framework for 360 panoramic view using latent space viewpoint estimation. We show that the proposed method is generic enough to achieve real-time rendering of arbitrary views for four datasets.
  arXiv  Detail & Related papers  (2024-01-07T09:17:32Z)
• PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation [39.269864548255576]
  We present a panoramic video dataset, PanoVOS. The dataset provides 150 videos with high video resolutions and diverse motions. We present a Panoramic Space Consistency Transformer (PSCFormer) which can effectively utilize the semantic boundary information of the previous frame for pixel-level matching with the current frame.
  arXiv  Detail & Related papers  (2023-09-21T17:59:02Z)
• 360-Degree Panorama Generation from Few Unregistered NFoV Images [16.05306624008911]
  360$circ$ panoramas are extensively utilized as environmental light sources in computer graphics. capturing a 360$circ$ $times$ 180$circ$ panorama poses challenges due to specialized and costly equipment. We propose a novel pipeline called PanoDiff, which efficiently generates complete 360$circ$ panoramas.
  arXiv  Detail & Related papers  (2023-08-28T16:21:51Z)
• NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes [59.15910989235392]
  We introduce NeO 360, Neural fields for sparse view synthesis of outdoor scenes. NeO 360 is a generalizable method that reconstructs 360deg scenes from a single or a few posed RGB images. Our representation combines the best of both voxel-based and bird's-eye-view (BEV) representations.
  arXiv  Detail & Related papers  (2023-08-24T17:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.