DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes

• URL: http://arxiv.org/abs/2412.11100v1
• Date: Sun, 15 Dec 2024 07:42:26 GMT
• Title: DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes
• Authors: Jinxiu Liu, Shaoheng Lin, Yinxiao Li, Ming-Hsuan Yang,
• Abstract summary: DynamicScaler enables spatially scalable and panoramic dynamic scene synthesis.<n>We employ a Global Motion Guidance mechanism to ensure both local detail fidelity and global motion continuity.<n>Our method achieves superior content and motion quality in panoramic scene-level video generation.
• Score: 46.91656616577897
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The increasing demand for immersive AR/VR applications and spatial intelligence has heightened the need to generate high-quality scene-level and 360{\deg} panoramic video. However, most video diffusion models are constrained by limited resolution and aspect ratio, which restricts their applicability to scene-level dynamic content synthesis. In this work, we propose the DynamicScaler, addressing these challenges by enabling spatially scalable and panoramic dynamic scene synthesis that preserves coherence across panoramic scenes of arbitrary size. Specifically, we introduce a Offset Shifting Denoiser, facilitating efficient, synchronous, and coherent denoising panoramic dynamic scenes via a diffusion model with fixed resolution through a seamless rotating Window, which ensures seamless boundary transitions and consistency across the entire panoramic space, accommodating varying resolutions and aspect ratios. Additionally, we employ a Global Motion Guidance mechanism to ensure both local detail fidelity and global motion continuity. Extensive experiments demonstrate our method achieves superior content and motion quality in panoramic scene-level video generation, offering a training-free, efficient, and scalable solution for immersive dynamic scene creation with constant VRAM consumption regardless of the output video resolution. Our project page is available at \url{https://dynamic-scaler.pages.dev/}.

Related papers

• M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation [65.08520614570288]
  We reformulate talking head generation into a unified framework comprising video preprocessing, motion representation, and rendering reconstruction.<n>M2DAO-Talker achieves state-of-the-art performance, with the 2.43 dB PSNR improvement in generation quality and 0.64 gain in user-evaluated video realness.
  arXiv  Detail & Related papers  (2025-07-11T04:48:12Z)
• LocalDyGS: Multi-view Global Dynamic Scene Modeling via Adaptive Local Implicit Feature Decoupling [33.71658540929536]
  LocalDyGS is a novel method to model dynamic videos from multi-view inputs for arbitrary viewpoints.<n>Our method is competitive on various fine-scale datasets compared to state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2025-07-03T06:50:33Z)
• PanoWan: Lifting Diffusion Video Generation Models to 360° with Latitude/Longitude-aware Mechanisms [41.92179513409301]
  Existing panoramic video generation models struggle to leverage pre-trained generative priors from conventional text-to-video models for high-quality panoramic videos.<n>In this paper, we introduce PanoWan to effectively lift pre-trained text-to-video models to the panoramic domain, equipped with minimal modules.<n>To provide sufficient panoramic videos for learning these lifted representations, we contribute PanoVid, a high-quality panoramic video dataset with captions and diverse scenarios.
  arXiv  Detail & Related papers  (2025-05-28T06:24:21Z)
• CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models [89.63787060844409]
  CameraCtrl II is a framework that enables large-scale dynamic scene exploration through a camera-controlled video diffusion model. We take an approach that progressively expands the generation of dynamic scenes.
  arXiv  Detail & Related papers  (2025-03-13T17:42:01Z)
• T-SVG: Text-Driven Stereoscopic Video Generation [87.62286959918566]
  This paper introduces the Text-driven Stereoscopic Video Generation (T-SVG) system. It streamlines video generation by using text prompts to create reference videos. These videos are transformed into 3D point cloud sequences, which are rendered from two perspectives with subtle parallax differences.
  arXiv  Detail & Related papers  (2024-12-12T14:48:46Z)
• DiffMVR: Diffusion-based Automated Multi-Guidance Video Restoration [10.637125300701795]
  We propose a diffusion-based video-level inpainting model, DiffMVR.<n>Our approach introduces a dynamic dual-guided image prompting system, leveraging adaptive reference frames to guide the inpainting process.<n>This enables the model to capture both fine-grained details and smooth transitions between video frames, offering precise control over inpainting direction and significantly improving restoration accuracy in challenging, dynamic environments.
  arXiv  Detail & Related papers  (2024-11-27T20:43:35Z)
• Modeling Ambient Scene Dynamics for Free-view Synthesis [31.233859111566613]
  We introduce a novel method for dynamic free-view synthesis of an ambient scenes from a monocular capture. Our method builds upon the recent advancements in 3D Gaussian Splatting (3DGS) that can faithfully reconstruct complex static scenes.
  arXiv  Detail & Related papers  (2024-06-13T17:59:11Z)
• Lumiere: A Space-Time Diffusion Model for Video Generation [75.54967294846686]
  We introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once. This is in contrast to existing video models which synthesize distants followed by temporal super-resolution. By deploying both spatial and (importantly) temporal down- and up-sampling, our model learns to directly generate a full-frame-rate, low-resolution video.
  arXiv  Detail & Related papers  (2024-01-23T18:05:25Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• Editable Free-viewpoint Video Using a Layered Neural Representation [35.44420164057911]
  We propose the first approach for editable free-viewpoint video generation for large-scale dynamic scenes using only sparse 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity including the environment itself is formulated into a space-time coherent neural layered radiance representation called ST-NeRF. Experiments demonstrate the effectiveness of our approach to achieve high-quality, photo-realistic, and editable free-viewpoint video generation for dynamic scenes.
  arXiv  Detail & Related papers  (2021-04-30T06:50:45Z)
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
  Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
  arXiv  Detail & Related papers  (2020-12-22T18:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.