CTRL-GS: Cascaded Temporal Residue Learning for 4D Gaussian Splatting

• URL: http://arxiv.org/abs/2505.18306v2
• Date: Sat, 31 May 2025 18:11:43 GMT
• Title: CTRL-GS: Cascaded Temporal Residue Learning for 4D Gaussian Splatting
• Authors: Karly Hou, Wanhua Li, Hanspeter Pfister,
• Abstract summary: We propose a novel extension to 4D Gaussian Splatting for dynamic scenes.<n>We decompose the dynamic scene into a "video-segment-frame" structure, with segments dynamically adjusted by optical flow.<n>We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.
• Score: 28.308077474731594
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, Gaussian Splatting methods have emerged as a desirable substitute for prior Radiance Field methods for novel-view synthesis of scenes captured with multi-view images or videos. In this work, we propose a novel extension to 4D Gaussian Splatting for dynamic scenes. Drawing on ideas from residual learning, we hierarchically decompose the dynamic scene into a "video-segment-frame" structure, with segments dynamically adjusted by optical flow. Then, instead of directly predicting the time-dependent signals, we model the signal as the sum of video-constant values, segment-constant values, and frame-specific residuals, as inspired by the success of residual learning. This approach allows more flexible models that adapt to highly variable scenes. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets, with the greatest improvements on complex scenes with large movements, occlusions, and fine details, where current methods degrade most.

Related papers

• VDEGaussian: Video Diffusion Enhanced 4D Gaussian Splatting for Dynamic Urban Scenes Modeling [68.65587507038539]
  We present a novel video diffusion-enhanced 4D Gaussian Splatting framework for dynamic urban scene modeling.<n>Our key insight is to distill robust, temporally consistent priors from a test-time adapted video diffusion model.<n>Our method significantly enhances dynamic modeling, especially for fast-moving objects, achieving an approximate PSNR gain of 2 dB.
  arXiv  Detail & Related papers  (2025-08-04T07:24:05Z)
• HoliGS: Holistic Gaussian Splatting for Embodied View Synthesis [59.25751939710903]
  We propose a novel deformable Gaussian splatting framework that addresses embodied view synthesis from long monocular RGB videos.<n>Our method leverages invertible Gaussian Splatting deformation networks to reconstruct large-scale, dynamic environments accurately.<n>Results highlight a practical and scalable solution for EVS in real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-24T03:54:40Z)
• Efficient 4D Gaussian Stream with Low Rank Adaptation [8.69899446610606]
  We propose a highly scalable method for dynamic novel view synthesis with continual learning.<n>Our method continuously reconstructs the dynamics with chunks of video frames, reduces the streaming bandwidth by $90%$ while maintaining high rendering quality comparable to the off-line SOTA methods.
  arXiv  Detail & Related papers  (2025-02-23T13:48:52Z)
• GaussianVideo: Efficient Video Representation via Hierarchical Gaussian Splatting [28.981174430968643]
  We introduce a novel neural representation that combines 3D Gaussian splatting with continuous camera motion modeling.<n> Experimental results show that our hierarchical learning, combined with robust camera motion modeling, captures complex dynamic scenes with strong temporal consistency.<n>This memory-efficient approach achieves high-quality rendering at impressive speeds.
  arXiv  Detail & Related papers  (2025-01-08T19:01:12Z)
• 4D Gaussian Splatting with Scale-aware Residual Field and Adaptive Optimization for Real-time Rendering of Temporally Complex Dynamic Scenes [19.24815625343669]
  SaRO-GS is a novel dynamic scene representation capable of achieving real-time rendering.<n>To handle temporally complex dynamic scenes, we introduce a Scale-aware Residual Field.<n>Our method has demonstrated state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-09T08:44:19Z)
• Adaptive and Temporally Consistent Gaussian Surfels for Multi-view Dynamic Reconstruction [3.9363268745580426]
  AT-GS is a novel method for reconstructing high-quality dynamic surfaces from multi-view videos through per-frame incremental optimization. We reduce temporal jittering in dynamic surfaces by ensuring consistency in curvature maps across consecutive frames. Our method achieves superior accuracy and temporal coherence in dynamic surface reconstruction, delivering high-fidelity space-time novel view synthesis.
  arXiv  Detail & Related papers  (2024-11-10T21:30:16Z)
• VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction [59.40711222096875]
  We present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting. Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets.
  arXiv  Detail & Related papers  (2024-02-27T11:40:50Z)
• SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting [7.553079256251747]
  We extend 3D Gaussian Splatting to reconstruct dynamic scenes. We produce high-quality renderings of general dynamic scenes with competitive quantitative performance. Our method can be viewed in real-time in our dynamic interactive viewer.
  arXiv  Detail & Related papers  (2023-12-20T03:54:03Z)
• Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis [58.5779956899918]
  We present a method that simultaneously addresses the tasks of dynamic scene novel-view synthesis and six degree-of-freedom (6-DOF) tracking of all dense scene elements. We follow an analysis-by-synthesis framework, inspired by recent work that models scenes as a collection of 3D Gaussians. We demonstrate a large number of downstream applications enabled by our representation, including first-person view synthesis, dynamic compositional scene synthesis, and 4D video editing.
  arXiv  Detail & Related papers  (2023-08-18T17:59:21Z)
• SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes [75.9110646062442]
  We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.
  arXiv  Detail & Related papers  (2023-08-16T09:50:35Z)
• Leaping Into Memories: Space-Time Deep Feature Synthesis [93.10032043225362]
  We propose LEAPS, an architecture-independent method for synthesizing videos from internal models. We quantitatively and qualitatively evaluate the applicability of LEAPS by inverting a range of architectures convolutional attention-based on Kinetics-400.
  arXiv  Detail & Related papers  (2023-03-17T12:55:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.