Variational Bayes Gaussian Splatting

• URL: http://arxiv.org/abs/2410.03592v1
• Date: Fri, 4 Oct 2024 16:52:03 GMT
• Title: Variational Bayes Gaussian Splatting
• Authors: Toon Van de Maele, Ozan Catal, Alexander Tschantz, Christopher L. Buckley, Tim Verbelen,
• Abstract summary: 3D Gaussian Splatting has emerged as a promising approach for modeling 3D scenes using mixtures of Gaussians. We propose Variational Bayes Gaussian Splatting, a novel approach that frames training a Gaussian splat as variational inference over model parameters. Our experiments show that VBGS not only matches state-of-the-art performance on static datasets, but also enables continual learning from sequentially streamed 2D and 3D data.
• Score: 44.43761190929142
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, 3D Gaussian Splatting has emerged as a promising approach for modeling 3D scenes using mixtures of Gaussians. The predominant optimization method for these models relies on backpropagating gradients through a differentiable rendering pipeline, which struggles with catastrophic forgetting when dealing with continuous streams of data. To address this limitation, we propose Variational Bayes Gaussian Splatting (VBGS), a novel approach that frames training a Gaussian splat as variational inference over model parameters. By leveraging the conjugacy properties of multivariate Gaussians, we derive a closed-form variational update rule, allowing efficient updates from partial, sequential observations without the need for replay buffers. Our experiments show that VBGS not only matches state-of-the-art performance on static datasets, but also enables continual learning from sequentially streamed 2D and 3D data, drastically improving performance in this setting.

Related papers

• GP-GS: Gaussian Processes for Enhanced Gaussian Splatting [10.45038376276218]
  This paper proposes a novel 3D reconstruction framework that achieves adaptive and uncertainty-guided densification of sparse SfM point clouds. The pipeline utilizes uncertainty estimates to guide the pruning of high-variance predictions. Experiments conducted on synthetic and real-world datasets validate the effectiveness and practicality of the proposed framework.
  arXiv  Detail & Related papers  (2025-02-04T12:50:16Z)
• Generative Densification: Learning to Densify Gaussians for High-Fidelity Generalizable 3D Reconstruction [6.273357335397336]
  We propose Generative Densification, an efficient and generalizable method to densify Gaussians generated by feed-forward models. We show that our method outperforms state-of-the-art approaches with comparable or smaller model sizes.
  arXiv  Detail & Related papers  (2024-12-09T06:20:51Z)
• DiffGS: Functional Gaussian Splatting Diffusion [33.07847512591061]
  3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity. However, the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. We propose DiffGS, a general Gaussian generator based on latent diffusion models.
  arXiv  Detail & Related papers  (2024-10-25T16:08:08Z)
• PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting [59.277480452459315]
  We propose a principled sensitivity pruning score that preserves visual fidelity and foreground details at significantly higher compression ratios. We also propose a multi-round prune-refine pipeline that can be applied to any pretrained 3D-GS model without changing its training pipeline.
  arXiv  Detail & Related papers  (2024-06-14T17:53:55Z)
• latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction [48.86083272054711]
  latentSplat is a method to predict semantic Gaussians in a 3D latent space that can be splatted and decoded by a light-weight generative 2D architecture. We show that latentSplat outperforms previous works in reconstruction quality and generalization, while being fast and scalable to high-resolution data.
  arXiv  Detail & Related papers  (2024-03-24T20:48:36Z)
• Mesh-based Gaussian Splatting for Real-time Large-scale Deformation [58.18290393082119]
  It is challenging for users to directly deform or manipulate implicit representations with large deformations in the real-time fashion. We develop a novel GS-based method that enables interactive deformation. Our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate.
  arXiv  Detail & Related papers  (2024-02-07T12:36:54Z)
• Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle [9.082693946898733]
  We introduce a novel point-based approach for fast dynamic scene reconstruction and real-time rendering from both multi-view and monocular videos. In contrast to the prevalent NeRF-based approaches hampered by slow training and rendering speeds, our approach harnesses recent advancements in point-based 3D Gaussian Splatting (3DGS) Our proposed approach showcases a substantial efficiency improvement, achieving a $5times$ faster training speed compared to the per-frame 3DGS modeling.
  arXiv  Detail & Related papers  (2023-12-06T11:25:52Z)
• GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis [70.24111297192057]
  We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner. The proposed method enables 2K-resolution rendering under a sparse-view camera setting.
  arXiv  Detail & Related papers  (2023-12-04T18:59:55Z)
• GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting [51.96353586773191]
  We introduce textbfGS-SLAM that first utilizes 3D Gaussian representation in the Simultaneous Localization and Mapping system. Our method utilizes a real-time differentiable splatting rendering pipeline that offers significant speedup to map optimization and RGB-D rendering. Our method achieves competitive performance compared with existing state-of-the-art real-time methods on the Replica, TUM-RGBD datasets.
  arXiv  Detail & Related papers  (2023-11-20T12:08:23Z)
• Generalised Gaussian Process Latent Variable Models (GPLVM) with Stochastic Variational Inference [9.468270453795409]
  We study the doubly formulation of the BayesianVM model amenable with minibatch training. We show how this framework is compatible with different latent variable formulations and perform experiments to compare a suite of models. We demonstrate how we can train in the presence of massively missing data and obtain high-fidelity reconstructions.
  arXiv  Detail & Related papers  (2022-02-25T21:21:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.