Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis

• URL: http://arxiv.org/abs/2406.06216v1
• Date: Mon, 10 Jun 2024 12:33:08 GMT
• Title: Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis
• Authors: Xin Jin, Pengyi Jiao, Zheng-Peng Duan, Xingchao Yang, Chun-Le Guo, Bo Ren, Chongyi Li,
• Abstract summary: Volumetric rendering based methods excel in HDR view synthesis from RAWimages, especially for nighttime scenes. They suffer from long training times and cannot perform real-time rendering due to dense sampling requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time rendering and faster training. These designs enable LE3D to perform real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes.
• Score: 35.81034036380374
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Volumetric rendering based methods, like NeRF, excel in HDR view synthesis from RAWimages, especially for nighttime scenes. While, they suffer from long training times and cannot perform real-time rendering due to dense sampling requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time rendering and faster training. However, implementing RAW image-based view synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1) in nighttime scenes, extremely low SNR leads to poor structure-from-motion (SfM) estimation in distant views; 2) the limited representation capacity of spherical harmonics (SH) function is unsuitable for RAW linear color space; and 3) inaccurate scene structure hampers downstream tasks such as refocusing. To address these issues, we propose LE3D (Lighting Every darkness with 3DGS). Our method proposes Cone Scatter Initialization to enrich the estimation of SfM, and replaces SH with a Color MLP to represent the RAW linear color space. Additionally, we introduce depth distortion and near-far regularizations to improve the accuracy of scene structure for downstream tasks. These designs enable LE3D to perform real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes. Compared to previous volumetric rendering based methods, LE3D reduces training time to 1% and improves rendering speed by up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can be found in https://github.com/Srameo/LE3D .

Related papers

• Speedy-Splat: Fast 3D Gaussian Splatting with Sparse Pixels and Sparse Primitives [60.217580865237835]
  3D Gaussian Splatting (3D-GS) is a recent 3D scene reconstruction technique that enables real-time rendering of novel views by modeling scenes as parametric point clouds of differentiable 3D Gaussians. We identify and address two key inefficiencies in 3D-GS to substantially improve rendering speed. Our Speedy-Splat approach combines these techniques to accelerate average rendering speed by a drastic $mathit6.71times$ across scenes from the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets.
  arXiv  Detail & Related papers  (2024-11-30T20:25:56Z)
• EVER: Exact Volumetric Ellipsoid Rendering for Real-time View Synthesis [72.53316783628803]
  We present Exact Volumetric Ellipsoid Rendering (EVER), a method for real-time differentiable emission-only volume rendering. Unlike recentization based approach by 3D Gaussian Splatting (3DGS), our primitive based representation allows for exact volume rendering. We show that our method is more accurate with blending issues than 3DGS and follow-up work on view rendering.
  arXiv  Detail & Related papers  (2024-10-02T17:59:09Z)
• HDRSplat: Gaussian Splatting for High Dynamic Range 3D Scene Reconstruction from Raw Images [14.332077246864628]
  3D Gaussian Splatting (3DGS) has revolutionized the 3D scene reconstruction space enabling high-fidelity novel view in real-time. However, prior 3DGS and NeRF-based methods rely on 8-bit tone-mapped Low Dynamic Range images for scene reconstruction. Our proposed method HDRSplat tailors 3DGS to train directly on 14-bit linear raw images in near darkness which preserves the scenes' full dynamic range and content.
  arXiv  Detail & Related papers  (2024-07-23T14:21:00Z)
• Gaussian Time Machine: A Real-Time Rendering Methodology for Time-Variant Appearances [10.614750331310804]
  We present Gaussian Time Machine (GTM) which models the time-dependent attributes of Gaussian primitives with discrete time embedding vectors decoded by a lightweight Multi-Layer-Perceptron(MLP) GTM achieved state-of-the-art rendering fidelity on 3 datasets and is 100 times faster than NeRF-based counterparts in rendering.
  arXiv  Detail & Related papers  (2024-05-22T14:40:42Z)
• Denoising Diffusion via Image-Based Rendering [54.20828696348574]
  We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
  arXiv  Detail & Related papers  (2024-02-05T19:00:45Z)
• Deblurring 3D Gaussian Splatting [7.315329140016319]
  We propose a novel real-time deblurring framework, Deblurring 3D Gaussian Splatting, using a small Multi-Layer Perceptron (MLP) While Deblurring 3D Gaussian Splatting can still enjoy real-time rendering, it can reconstruct fine and sharp details from blurry images.
  arXiv  Detail & Related papers  (2024-01-01T18:23:51Z)
• SparseGS: Real-Time 360° Sparse View Synthesis using Gaussian Splatting [6.506706621221143]
  3D Splatting (3DGS) has recently enabled real-time rendering of 3D scenes for novel view synthesis. This technique requires dense training views to accurately reconstruct 3D geometry. We introduce SparseGS, an efficient training pipeline designed to address the limitations of 3DGS in scenarios with sparse training views.
  arXiv  Detail & Related papers  (2023-11-30T21:38:22Z)
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering [4.320393382724066]
  We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.
  arXiv  Detail & Related papers  (2023-08-08T06:37:06Z)
• Real-Time Radiance Fields for Single-Image Portrait View Synthesis [85.32826349697972]
  We present a one-shot method to infer and render a 3D representation from a single unposed image in real-time. Given a single RGB input, our image encoder directly predicts a canonical triplane representation of a neural radiance field for 3D-aware novel view synthesis via volume rendering. Our method is fast (24 fps) on consumer hardware, and produces higher quality results than strong GAN-inversion baselines that require test-time optimization.
  arXiv  Detail & Related papers  (2023-05-03T17:56:01Z)
• 3D Scene Creation and Rendering via Rough Meshes: A Lighting Transfer Avenue [49.62477229140788]
  This paper studies how to flexibly integrate reconstructed 3D models into practical 3D modeling pipelines such as 3D scene creation and rendering. We propose a lighting transfer network (LighTNet) to bridge NFR and PBR, such that they can benefit from each other.
  arXiv  Detail & Related papers  (2022-11-27T13:31:00Z)
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis [163.96778522283967]
  We propose a shading-guided generative implicit model that is able to learn a starkly improved shape representation. An accurate 3D shape should also yield a realistic rendering under different lighting conditions. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2021-10-29T10:53:12Z)
• Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
  We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
  arXiv  Detail & Related papers  (2021-09-13T15:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.