360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model

• URL: http://arxiv.org/abs/2401.06578v2
• Date: Fri, 10 May 2024 12:11:16 GMT
• Title: 360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model
• Authors: Qian Wang, Weiqi Li, Chong Mou, Xinhua Cheng, Jian Zhang,
• Abstract summary: We propose a pipeline named 360-Degree Video Diffusion model (360DVD) for generating 360-degree panoramic videos. We introduce a lightweight 360-Adapter accompanied by 360 Enhancement Techniques to transform pre-trained T2V models for panorama video generation. We also propose a new panorama dataset named WEB360 consisting of panoramic video-text pairs for training 360DVD.
• Score: 23.708946172342067
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Panorama video recently attracts more interest in both study and application, courtesy of its immersive experience. Due to the expensive cost of capturing 360-degree panoramic videos, generating desirable panorama videos by prompts is urgently required. Lately, the emerging text-to-video (T2V) diffusion methods demonstrate notable effectiveness in standard video generation. However, due to the significant gap in content and motion patterns between panoramic and standard videos, these methods encounter challenges in yielding satisfactory 360-degree panoramic videos. In this paper, we propose a pipeline named 360-Degree Video Diffusion model (360DVD) for generating 360-degree panoramic videos based on the given prompts and motion conditions. Specifically, we introduce a lightweight 360-Adapter accompanied by 360 Enhancement Techniques to transform pre-trained T2V models for panorama video generation. We further propose a new panorama dataset named WEB360 consisting of panoramic video-text pairs for training 360DVD, addressing the absence of captioned panoramic video datasets. Extensive experiments demonstrate the superiority and effectiveness of 360DVD for panorama video generation. Our project page is at https://akaneqwq.github.io/360DVD/.

Related papers

• VideoPanda: Video Panoramic Diffusion with Multi-view Attention [57.87428280844657]
  High resolution panoramic video content is paramount for immersive experiences in Virtual Reality, but is non-trivial to collect as it requires specialized equipment and intricate camera setups. VideoPanda generates more realistic and coherent 360$circ$ panoramas across all input conditions compared to existing methods.
  arXiv  Detail & Related papers  (2025-04-15T16:58:15Z)
• Beyond the Frame: Generating 360° Panoramic Videos from Perspective Videos [64.10180665546237]
  360deg videos offer a more complete perspective of our surroundings. Existing video models excel at producing standard videos, but their ability to generate full panoramic videos remains elusive. We develop a high-quality data filtering pipeline to curate pairwise training data and improve the quality of 360deg video generation. Experimental results demonstrate that our model can generate realistic and coherent 360deg videos from in-the-wild perspective video.
  arXiv  Detail & Related papers  (2025-04-10T17:51:38Z)
• WorldPrompter: Traversable Text-to-Scene Generation [18.405299478122693]
  We introduce WorldPrompter, a novel generative pipeline for synthesizing traversable 3D scenes from text prompts. WorldPrompter incorporates a conditional 360deg panoramic video generator, capable of producing a 128-frame video that simulates a person walking through and capturing a virtual environment. The resulting video is then reconstructed as Gaussian splats by a fast feedforward 3D reconstructor, enabling a true walkable experience within the 3D scene.
  arXiv  Detail & Related papers  (2025-04-02T18:04:32Z)
• Imagine360: Immersive 360 Video Generation from Perspective Anchor [79.97844408255897]
  Imagine360 is a perspective-to-$360circ$ video generation framework. It learns fine-grained spherical visual and motion patterns from limited $360circ$ video data. It achieves superior graphics quality and motion coherence among state-of-the-art $360circ$ video generation methods.
  arXiv  Detail & Related papers  (2024-12-04T18:50:08Z)
• DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion [60.45000652592418]
  We propose a novel text-driven panoramic generation framework, DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. We show that DiffPano can generate consistent, diverse panoramic images with given unseen text descriptions and camera poses.
  arXiv  Detail & Related papers  (2024-10-31T17:57:02Z)
• VidPanos: Generative Panoramic Videos from Casual Panning Videos [73.77443496436749]
  Panoramic image stitching provides a unified, wide-angle view of a scene that extends beyond the camera's field of view. We present a method for synthesizing a panoramic video from a casually-captured panning video. Our system can create video panoramas for a range of in-the-wild scenes including people, vehicles, and flowing water.
  arXiv  Detail & Related papers  (2024-10-17T17:53:24Z)
• SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting [53.32467009064287]
  We propose a text-driven 3D-consistent scene generation model: SceneDreamer360. Our proposed method leverages a text-driven panoramic image generation model as a prior for 3D scene generation. Our experiments demonstrate that SceneDreamer360 with its panoramic image generation and 3DGS can produce higher quality, spatially consistent, and visually appealing 3D scenes from any text prompt.
  arXiv  Detail & Related papers  (2024-08-25T02:56:26Z)
• LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation [105.52153675890408]
  3D immersive scene generation is a challenging yet critical task in computer vision and graphics. Layerpano3D is a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt.
  arXiv  Detail & Related papers  (2024-08-23T17:50:23Z)
• SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix [60.48666051245761]
  We propose a pose-free and training-free approach for generating 3D stereoscopic videos. Our method warps a generated monocular video into camera views on stereoscopic baseline using estimated video depth. We develop a disocclusion boundary re-injection scheme that further improves the quality of video inpainting.
  arXiv  Detail & Related papers  (2024-06-29T08:33:55Z)
• 4K4DGen: Panoramic 4D Generation at 4K Resolution [67.98105958108503]
  We tackle the challenging task of elevating a single panorama to an immersive 4D experience. For the first time, we demonstrate the capability to generate omnidirectional dynamic scenes with 360$circ$ views at 4K resolution. We achieve high-quality Panorama-to-4D generation at a resolution of 4K for the first time.
  arXiv  Detail & Related papers  (2024-06-19T13:11:02Z)
• See360: Novel Panoramic View Interpolation [24.965259708297932]
  See360 is a versatile and efficient framework for 360 panoramic view using latent space viewpoint estimation. We show that the proposed method is generic enough to achieve real-time rendering of arbitrary views for four datasets.
  arXiv  Detail & Related papers  (2024-01-07T09:17:32Z)
• PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation [39.269864548255576]
  We present a panoramic video dataset, PanoVOS. The dataset provides 150 videos with high video resolutions and diverse motions. We present a Panoramic Space Consistency Transformer (PSCFormer) which can effectively utilize the semantic boundary information of the previous frame for pixel-level matching with the current frame.
  arXiv  Detail & Related papers  (2023-09-21T17:59:02Z)
• 360-Degree Panorama Generation from Few Unregistered NFoV Images [16.05306624008911]
  360$circ$ panoramas are extensively utilized as environmental light sources in computer graphics. capturing a 360$circ$ $times$ 180$circ$ panorama poses challenges due to specialized and costly equipment. We propose a novel pipeline called PanoDiff, which efficiently generates complete 360$circ$ panoramas.
  arXiv  Detail & Related papers  (2023-08-28T16:21:51Z)
• NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes [59.15910989235392]
  We introduce NeO 360, Neural fields for sparse view synthesis of outdoor scenes. NeO 360 is a generalizable method that reconstructs 360deg scenes from a single or a few posed RGB images. Our representation combines the best of both voxel-based and bird's-eye-view (BEV) representations.
  arXiv  Detail & Related papers  (2023-08-24T17:59:50Z)
• Revisiting Optical Flow Estimation in 360 Videos [9.997208301312956]
  We design LiteFlowNet360 as a domain adaptation framework from perspective video domain to 360 video domain. We adapt it from simple kernel transformation techniques inspired by Kernel Transformer Network (KTN) to cope with inherent distortion in 360 videos. Experimental results show the promising results of 360 video optical flow estimation using the proposed novel architecture.
  arXiv  Detail & Related papers  (2020-10-15T22:22:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.