From Chaos to Clarity: 3DGS in the Dark

• URL: http://arxiv.org/abs/2406.08300v1
• Date: Wed, 12 Jun 2024 15:00:16 GMT
• Title: From Chaos to Clarity: 3DGS in the Dark
• Authors: Zhihao Li, Yufei Wang, Alex Kot, Bihan Wen,
• Abstract summary: Noise in unprocessed raw images compromises accuracy of 3D scene representation. 3D Gaussian Splatting (3DGS) is particularly susceptible to this noise. We introduce a novel self-supervised learning framework designed to reconstruct HDR 3DGS from noisy raw images.
• Score: 28.232432162734437
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Novel view synthesis from raw images provides superior high dynamic range (HDR) information compared to reconstructions from low dynamic range RGB images. However, the inherent noise in unprocessed raw images compromises the accuracy of 3D scene representation. Our study reveals that 3D Gaussian Splatting (3DGS) is particularly susceptible to this noise, leading to numerous elongated Gaussian shapes that overfit the noise, thereby significantly degrading reconstruction quality and reducing inference speed, especially in scenarios with limited views. To address these issues, we introduce a novel self-supervised learning framework designed to reconstruct HDR 3DGS from a limited number of noisy raw images. This framework enhances 3DGS by integrating a noise extractor and employing a noise-robust reconstruction loss that leverages a noise distribution prior. Experimental results show that our method outperforms LDR/HDR 3DGS and previous state-of-the-art (SOTA) self-supervised and supervised pre-trained models in both reconstruction quality and inference speed on the RawNeRF dataset across a broad range of training views. Code can be found in \url{https://lizhihao6.github.io/Raw3DGS}.

Related papers

• NovelGS: Consistent Novel-view Denoising via Large Gaussian Reconstruction Model [57.92709692193132]
  NovelGS is a diffusion model for Gaussian Splatting given sparse-view images. We leverage the novel view denoising through a transformer-based network to generate 3D Gaussians.
  arXiv  Detail & Related papers  (2024-11-25T07:57:17Z)
• Beyond Gaussians: Fast and High-Fidelity 3D Splatting with Linear Kernels [51.08794269211701]
  We introduce 3D Linear Splatting (3DLS), which replaces Gaussian kernels with linear kernels to achieve sharper and more precise results. 3DLS demonstrates state-of-the-art fidelity and accuracy, along with a 30% FPS improvement over baseline 3DGS.
  arXiv  Detail & Related papers  (2024-11-19T11:59:54Z)
• Towards Degradation-Robust Reconstruction in Generalizable NeRF [58.33351079982745]
  Generalizable Radiance Field (GNeRF) across scenes has been proven to be an effective way to avoid per-scene optimization. There has been limited research on the robustness of GNeRFs to different types of degradation present in the source images.
  arXiv  Detail & Related papers  (2024-11-18T16:13:47Z)
• USP-Gaussian: Unifying Spike-based Image Reconstruction, Pose Correction and Gaussian Splatting [45.246178004823534]
  Spike cameras, as an innovative neuromorphic camera that captures scenes with the 0-1 bit stream at 40 kHz, are increasingly employed for the 3D reconstruction task. Previous spike-based 3D reconstruction approaches often employ a casecased pipeline. We propose a synergistic optimization framework, textbfUSP-Gaussian, that unifies spike-based image reconstruction, pose correction, and Gaussian splatting into an end-to-end framework.
  arXiv  Detail & Related papers  (2024-11-15T14:15:16Z)
• PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting [54.7468067660037]
  PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS.
  arXiv  Detail & Related papers  (2024-10-29T15:28:15Z)
• LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priors [34.91966359570867]
  sparse-view reconstruction is inherently ill-posed and under-constrained. We introduce LM-Gaussian, a method capable of generating high-quality reconstructions from a limited number of images. Our approach significantly reduces the data acquisition requirements compared to previous 3DGS methods.
  arXiv  Detail & Related papers  (2024-09-05T12:09:02Z)
• Splatfacto-W: A Nerfstudio Implementation of Gaussian Splatting for Unconstrained Photo Collections [25.154665328053333]
  We introduce Splatfacto-W, an in-trivial approach that integrates per-Gaussian neural color features and per-image appearance embeddings into an rendering process. Our method improves the Peak Signal-to-Noise Ratio (PSNR) by an average of 5.3 dB compared to 3DGS, enhances training speed by 150 times compared to NeRF-based methods, and achieves a similar rendering speed to 3DGS.
  arXiv  Detail & Related papers  (2024-07-17T04:02:54Z)
• WildGaussians: 3D Gaussian Splatting in the Wild [80.5209105383932]
  We introduce WildGaussians, a novel approach to handle occlusions and appearance changes with 3DGS. We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines in handling in-the-wild data.
  arXiv  Detail & Related papers  (2024-07-11T12:41:32Z)
• A Refined 3D Gaussian Representation for High-Quality Dynamic Scene Reconstruction [2.022451212187598]
  In recent years, Neural Radiance Fields (NeRF) has revolutionized three-dimensional (3D) reconstruction with its implicit representation. 3D Gaussian Splatting (3D-GS) has departed from the implicit representation of neural networks and instead directly represents scenes as point clouds with Gaussian-shaped distributions. This paper purposes a refined 3D Gaussian representation for high-quality dynamic scene reconstruction. Experimental results demonstrate that our method surpasses existing approaches in rendering quality and speed, while significantly reducing the memory usage associated with 3D-GS.
  arXiv  Detail & Related papers  (2024-05-28T07:12:22Z)
• GaussianDiffusion: 3D Gaussian Splatting for Denoising Diffusion Probabilistic Models with Structured Noise [0.0]
  This paper introduces a novel text to 3D content generation framework based on Gaussian splatting. The challenge of achieving multi-view consistency in 3D generation significantly impedes modeling complexity and accuracy.
  arXiv  Detail & Related papers  (2023-11-19T04:26:16Z)
• Towards High-quality HDR Deghosting with Conditional Diffusion Models [88.83729417524823]
  High Dynamic Range (LDR) images can be recovered from several Low Dynamic Range (LDR) images by existing Deep Neural Networks (DNNs) techniques. DNNs still generate ghosting artifacts when LDR images have saturation and large motion. We formulate the HDR deghosting problem as an image generation that leverages LDR features as the diffusion model's condition.
  arXiv  Detail & Related papers  (2023-11-02T01:53:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.