4D Scaffold Gaussian Splatting for Memory Efficient Dynamic Scene Reconstruction

• URL: http://arxiv.org/abs/2411.17044v1
• Date: Tue, 26 Nov 2024 02:22:07 GMT
• Title: 4D Scaffold Gaussian Splatting for Memory Efficient Dynamic Scene Reconstruction
• Authors: Woong Oh Cho, In Cho, Seoha Kim, Jeongmin Bae, Youngjung Uh, Seon Joo Kim,
• Abstract summary: This paper proposes a 4D anchor-based framework that retains visual quality and rendering speed of 4D Gaussians while reducing storage costs. Experimental results demonstrate that our method achieves state-of-the-art visual quality and 97.8% storage reduction over 4DGS.
• Score: 27.455934322535853
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing 4D Gaussian methods for dynamic scene reconstruction offer high visual fidelity and fast rendering. However, these methods suffer from excessive memory and storage demands, which limits their practical deployment. This paper proposes a 4D anchor-based framework that retains visual quality and rendering speed of 4D Gaussians while significantly reducing storage costs. Our method extends 3D scaffolding to 4D space, and leverages sparse 4D grid-aligned anchors with compressed feature vectors. Each anchor models a set of neural 4D Gaussians, each of which represent a local spatiotemporal region. In addition, we introduce a temporal coverage-aware anchor growing strategy to effectively assign additional anchors to under-reconstructed dynamic regions. Our method adjusts the accumulated gradients based on Gaussians' temporal coverage, improving reconstruction quality in dynamic regions. To reduce the number of anchors, we further present enhanced formulations of neural 4D Gaussians. These include the neural velocity, and the temporal opacity derived from a generalized Gaussian distribution. Experimental results demonstrate that our method achieves state-of-the-art visual quality and 97.8% storage reduction over 4DGS.

Related papers

• VDEGaussian: Video Diffusion Enhanced 4D Gaussian Splatting for Dynamic Urban Scenes Modeling [68.65587507038539]
  We present a novel video diffusion-enhanced 4D Gaussian Splatting framework for dynamic urban scene modeling.<n>Our key insight is to distill robust, temporally consistent priors from a test-time adapted video diffusion model.<n>Our method significantly enhances dynamic modeling, especially for fast-moving objects, achieving an approximate PSNR gain of 2 dB.
  arXiv  Detail & Related papers  (2025-08-04T07:24:05Z)
• SD-GS: Structured Deformable 3D Gaussians for Efficient Dynamic Scene Reconstruction [5.818188539758898]
  We present SD-GS, a compact and efficient dynamic splatting framework for complex dynamic scene reconstruction.<n>We also present a deformation-aware densification strategy that adaptively grows anchors in under-reconstructed high-dynamic regions.<n> Experimental results demonstrate that SD-GS achieves an average of 60% reduction in model size and an average of 100% improvement in FPS.
  arXiv  Detail & Related papers  (2025-07-10T06:35:03Z)
• Speedy Deformable 3D Gaussian Splatting: Fast Rendering and Compression of Dynamic Scenes [57.69608119350651]
  Recent extensions of 3D Gaussian Splatting (3DGS) to dynamic scenes achieve high-quality novel view synthesis by using neural networks to predict the time-varying deformation of each Gaussian.<n>However, performing per-Gaussian neural inference at every frame poses a significant bottleneck, limiting rendering speed and increasing memory and compute requirements.<n>We present Speedy Deformable 3D Gaussian Splatting (SpeeDe3DGS), a general pipeline for accelerating the rendering speed of dynamic 3DGS and 4DGS representations by reducing neural inference through two complementary techniques.
  arXiv  Detail & Related papers  (2025-06-09T16:30:48Z)
• Hybrid 3D-4D Gaussian Splatting for Fast Dynamic Scene Representation [2.7463268699570134]
  4D Gaussian Splatting (4DGS) has emerged as an appealing approach due to its ability to model high-fidelity spatial and temporal variations.<n>We introduce hybrid 3D-4D Gaussian Splatting (3D-4DGS), a novel framework that adaptively represents static regions with 3D Gaussians while reserving 4D Gaussians for dynamic elements.<n>Our method achieves significantly faster training times compared to baseline 4D Gaussian Splatting methods while maintaining or improving the visual quality.
  arXiv  Detail & Related papers  (2025-05-19T14:59:58Z)
• EA-3DGS: Efficient and Adaptive 3D Gaussians with Highly Enhanced Quality for outdoor scenes [11.58555191379573]
  3D Gaussian Splatting (3DGS) has demonstrated excellent performance with its high-quality rendering and real-time speed.<n>We propose EA-3DGS, a high-quality real-time rendering method designed for outdoor scenes.
  arXiv  Detail & Related papers  (2025-05-16T02:00:13Z)
• Disentangled 4D Gaussian Splatting: Towards Faster and More Efficient Dynamic Scene Rendering [12.27734287104036]
  Novel-entangleview synthesis (NVS) for dynamic scenes from 2D images presents significant challenges. We introduce Disentangled 4D Gaussianting (Disentangled4DGS), a novel representation and rendering approach that disentangles temporal and spatial deformations. Our approach achieves an unprecedented average rendering speed of 343 FPS at a resolution of $1352times1014$ on a 3090 GPU.
  arXiv  Detail & Related papers  (2025-03-28T05:46:02Z)
• 4DGC: Rate-Aware 4D Gaussian Compression for Efficient Streamable Free-Viewpoint Video [56.04182926886754]
  3D Gaussian Splatting (3DGS) has substantial potential for enabling photorealistic Free-Viewpoint Video (FVV) experiences. Existing methods typically handle dynamic 3DGS representation and compression separately, motion information and the rate-distortion trade-off during training. We propose 4DGC, a rate-aware 4D Gaussian compression framework that significantly reduces storage size while maintaining superior RD performance for FVV.
  arXiv  Detail & Related papers  (2025-03-24T08:05:27Z)
• 1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering [60.676919690136096]
  We present textbf4DGS-1K, which runs at over 1000 FPS on modern scene GPUs. For Q1, we introduce the Spatial-Temporal Variation Score, a new pruning criterion that effectively removes short-lifespan Gaussians. For Q2, we store a mask for active Gaussians across consecutive frames, significantly reducing redundant computations in rendering.
  arXiv  Detail & Related papers  (2025-03-20T17:59:44Z)
• Efficient Gaussian Splatting for Monocular Dynamic Scene Rendering via Sparse Time-Variant Attribute Modeling [64.84686527988809]
  Deformable Gaussian Splatting has emerged as a robust solution to represent real-world dynamic scenes.<n>Our approach formulates dynamic scenes using a sparse anchor-grid representation, with the motion flow of dense Gaussians calculated via a classical kernel representation.<n>Experiments on two real-world datasets demonstrate that our EDGS significantly improves the rendering speed with superior rendering quality.
  arXiv  Detail & Related papers  (2025-02-27T18:53:06Z)
• 4DRGS: 4D Radiative Gaussian Splatting for Efficient 3D Vessel Reconstruction from Sparse-View Dynamic DSA Images [49.170407434313475]
  Existing methods often produce suboptimal results or require excessive computation time. We propose 4D radiative Gaussian splatting (4DRGS) to achieve high-quality reconstruction efficiently. 4DRGS achieves impressive results in 5 minutes training, which is 32x faster than the state-of-the-art method.
  arXiv  Detail & Related papers  (2024-12-17T13:51:56Z)
• Event-boosted Deformable 3D Gaussians for Dynamic Scene Reconstruction [50.873820265165975]
  We introduce the first approach combining event cameras, which capture high-temporal-resolution, continuous motion data, with deformable 3D-GS for dynamic scene reconstruction.<n>We propose a GS-Threshold Joint Modeling strategy, creating a mutually reinforcing process that greatly improves both 3D reconstruction and threshold modeling.<n>We contribute the first event-inclusive 4D benchmark with synthetic and real-world dynamic scenes, on which our method achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-11-25T08:23:38Z)
• Dynamics-Aware Gaussian Splatting Streaming Towards Fast On-the-Fly Training for 4D Reconstruction [12.111389926333592]
  Current 3DGS-based streaming methods treat the Gaussian primitives uniformly and constantly renew the densified Gaussians. We propose a novel three-stage pipeline for iterative streamable 4D dynamic spatial reconstruction. Our method achieves state-of-the-art performance in online 4D reconstruction, demonstrating a 20% improvement in on-the-fly training speed, superior representation quality, and real-time rendering capability.
  arXiv  Detail & Related papers  (2024-11-22T10:47:47Z)
• DGTR: Distributed Gaussian Turbo-Reconstruction for Sparse-View Vast Scenes [81.56206845824572]
  Novel-view synthesis (NVS) approaches play a critical role in vast scene reconstruction. Few-shot methods often struggle with poor reconstruction quality in vast environments. This paper presents DGTR, a novel distributed framework for efficient Gaussian reconstruction for sparse-view vast scenes.
  arXiv  Detail & Related papers  (2024-11-19T07:51:44Z)
• MEGA: Memory-Efficient 4D Gaussian Splatting for Dynamic Scenes [49.36091070642661]
  This paper introduces a memory-efficient framework for 4DGS. It achieves a storage reduction by approximately 190$times$ and 125$times$ on the Technicolor and Neural 3D Video datasets. It maintains comparable rendering speeds and scene representation quality, setting a new standard in the field.
  arXiv  Detail & Related papers  (2024-10-17T14:47:08Z)
• S4D: Streaming 4D Real-World Reconstruction with Gaussians and 3D Control Points [30.46796069720543]
  We introduce a novel approach for streaming 4D real-world reconstruction utilizing discrete 3D control points. This method physically models local rays and establishes a motion-decoupling coordinate system. By effectively merging traditional graphics with learnable pipelines, it provides a robust and efficient local 6-degrees-of-freedom (6 DoF) motion representation.
  arXiv  Detail & Related papers  (2024-08-23T12:51:49Z)
• Compact 3D Gaussian Splatting for Static and Dynamic Radiance Fields [13.729716867839509]
  We propose a learnable mask strategy that significantly reduces the number of Gaussians while preserving high performance. In addition, we propose a compact but effective representation of view-dependent color by employing a grid-based neural field. Our work provides a comprehensive framework for 3D scene representation, achieving high performance, fast training, compactness, and real-time rendering.
  arXiv  Detail & Related papers  (2024-08-07T14:56:34Z)
• LGS: A Light-weight 4D Gaussian Splatting for Efficient Surgical Scene Reconstruction [33.794584735264884]
  We introduce a Lightweight 4D Gaussian Splatting framework (LGS) for dynamic endoscopic reconstruction. To minimize the redundancy of Gaussian quantities, we propose Deformation-Aware Pruning. We also simplify the representation of textures and lighting in non-crucial areas by pruning the dimensions of Gaussian attributes.
  arXiv  Detail & Related papers  (2024-06-23T10:49:39Z)
• PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting [59.277480452459315]
  We propose a principled spatial sensitivity pruning score that outperforms current approaches. We also propose a multi-round prune-refine pipeline that can be applied to any pretrained 3D-GS model. Our pipeline increases the average rendering speed of 3D-GS by 2.65$times$ while retaining more salient foreground information.
  arXiv  Detail & Related papers  (2024-06-14T17:53:55Z)
• A Refined 3D Gaussian Representation for High-Quality Dynamic Scene Reconstruction [2.022451212187598]
  In recent years, Neural Radiance Fields (NeRF) has revolutionized three-dimensional (3D) reconstruction with its implicit representation. 3D Gaussian Splatting (3D-GS) has departed from the implicit representation of neural networks and instead directly represents scenes as point clouds with Gaussian-shaped distributions. This paper purposes a refined 3D Gaussian representation for high-quality dynamic scene reconstruction. Experimental results demonstrate that our method surpasses existing approaches in rendering quality and speed, while significantly reducing the memory usage associated with 3D-GS.
  arXiv  Detail & Related papers  (2024-05-28T07:12:22Z)
• SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer [57.506654943449796]
  We propose an efficient, sparse-controlled video-to-4D framework named SC4D that decouples motion and appearance. Our method surpasses existing methods in both quality and efficiency. We devise a novel application that seamlessly transfers motion onto a diverse array of 4D entities.
  arXiv  Detail & Related papers  (2024-04-04T18:05:18Z)
• DreamGaussian4D: Generative 4D Gaussian Splatting [56.49043443452339]
  We introduce DreamGaussian4D (DG4D), an efficient 4D generation framework that builds on Gaussian Splatting (GS) Our key insight is that combining explicit modeling of spatial transformations with static GS makes an efficient and powerful representation for 4D generation. Video generation methods have the potential to offer valuable spatial-temporal priors, enhancing the high-quality 4D generation.
  arXiv  Detail & Related papers  (2023-12-28T17:16:44Z)
• Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering [71.44349029439944]
  Recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering.
  arXiv  Detail & Related papers  (2023-11-30T17:58:57Z)
• 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering [103.32717396287751]
  We propose 4D Gaussian Splatting (4D-GS) as a holistic representation for dynamic scenes. A neuralvoxel encoding algorithm inspired by HexPlane is proposed to efficiently build features from 4D neural voxels. Our 4D-GS method achieves real-time rendering under high resolutions, 82 FPS at an 800$times$800 resolution on an 3090 GPU.
  arXiv  Detail & Related papers  (2023-10-12T17:21:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.