HDR-GS: Efficient High Dynamic Range Novel View Synthesis at 1000x Speed via Gaussian Splatting

• URL: http://arxiv.org/abs/2405.15125v4
• Date: Sat, 26 Oct 2024 15:46:05 GMT
• Title: HDR-GS: Efficient High Dynamic Range Novel View Synthesis at 1000x Speed via Gaussian Splatting
• Authors: Yuanhao Cai, Zihao Xiao, Yixun Liang, Minghan Qin, Yulun Zhang, Xiaokang Yang, Yaoyao Liu, Alan Yuille,
• Abstract summary: Existing HDR NVS methods are mainly based on NeRF. They suffer from long training time and slow inference speed. We propose a new framework, High Dynamic Range Gaussian Splatting (-GS)
• Score: 76.5908492298286
• License:
• Abstract: High dynamic range (HDR) novel view synthesis (NVS) aims to create photorealistic images from novel viewpoints using HDR imaging techniques. The rendered HDR images capture a wider range of brightness levels containing more details of the scene than normal low dynamic range (LDR) images. Existing HDR NVS methods are mainly based on NeRF. They suffer from long training time and slow inference speed. In this paper, we propose a new framework, High Dynamic Range Gaussian Splatting (HDR-GS), which can efficiently render novel HDR views and reconstruct LDR images with a user input exposure time. Specifically, we design a Dual Dynamic Range (DDR) Gaussian point cloud model that uses spherical harmonics to fit HDR color and employs an MLP-based tone-mapper to render LDR color. The HDR and LDR colors are then fed into two Parallel Differentiable Rasterization (PDR) processes to reconstruct HDR and LDR views. To establish the data foundation for the research of 3D Gaussian splatting-based methods in HDR NVS, we recalibrate the camera parameters and compute the initial positions for Gaussian point clouds. Experiments demonstrate that our HDR-GS surpasses the state-of-the-art NeRF-based method by 3.84 and 1.91 dB on LDR and HDR NVS while enjoying 1000x inference speed and only requiring 6.3% training time. Code and recalibrated data will be publicly available at https://github.com/caiyuanhao1998/HDR-GS . A brief video introduction of our work is available at https://youtu.be/wtU7Kcwe7ck

Related papers

• Diffusion-Promoted HDR Video Reconstruction [45.73396977607666]
  High dynamic range (LDR) video reconstruction aims to generate HDR videos from low dynamic range (LDR) frames captured with alternating exposures. Most existing works solely rely on the regression-based paradigm, leading to adverse effects such as ghosting artifacts and missing details in saturated regions. We propose a diffusion-promoted method for HDR video reconstruction, termed HDR-V-Diff, which incorporates a diffusion model to capture the HDR distribution.
  arXiv  Detail & Related papers  (2024-06-12T13:38:10Z)
• Fast High Dynamic Range Radiance Fields for Dynamic Scenes [39.3304365600248]
  We propose a dynamic HDR NeRF framework, named HDR-HexPlane, which can learn 3D scenes from dynamic 2D images captured with various exposures. With the proposed model, high-quality novel-view images at any time point can be rendered with any desired exposure.
  arXiv  Detail & Related papers  (2024-01-11T17:15:16Z)
• Pano-NeRF: Synthesizing High Dynamic Range Novel Views with Geometry from Sparse Low Dynamic Range Panoramic Images [82.1477261107279]
  We propose the irradiance fields from sparse LDR panoramic images to increase the observation counts for faithful geometry recovery. Experiments demonstrate that the irradiance fields outperform state-of-the-art methods on both geometry recovery and HDR reconstruction.
  arXiv  Detail & Related papers  (2023-12-26T08:10:22Z)
• Efficient HDR Reconstruction from Real-World Raw Images [16.54071503000866]
  High-definition screens on edge devices stimulate a strong demand for efficient high dynamic range ( HDR) algorithms. Many existing HDR methods either deliver unsatisfactory results or consume too much computational and memory resources. In this work, we discover an excellent opportunity for HDR reconstructing directly from raw images and investigating novel neural network structures.
  arXiv  Detail & Related papers  (2023-06-17T10:10:15Z)
• HDR Video Reconstruction with a Large Dynamic Dataset in Raw and sRGB Domains [23.309488653045026]
  High dynamic range ( HDR) video reconstruction is attracting more and more attention due to the superior visual quality compared with those of low dynamic range (LDR) videos. There are still no real LDR- pairs for dynamic scenes due to the difficulty in capturing LDR- frames simultaneously. In this work, we propose to utilize a staggered sensor to capture two alternate exposure images simultaneously, which are then fused into an HDR frame in both raw and sRGB domains.
  arXiv  Detail & Related papers  (2023-04-10T11:59:03Z)
• GlowGAN: Unsupervised Learning of HDR Images from LDR Images in the Wild [74.52723408793648]
  We present the first method for learning a generative model of HDR images from in-the-wild LDR image collections in a fully unsupervised manner. The key idea is to train a generative adversarial network (GAN) to generate HDR images which, when projected to LDR under various exposures, are indistinguishable from real LDR images. Experiments show that our method GlowGAN can synthesize photorealistic HDR images in many challenging cases such as landscapes, lightning, or windows.
  arXiv  Detail & Related papers  (2022-11-22T15:42:08Z)
• HDR-NeRF: High Dynamic Range Neural Radiance Fields [70.80920996881113]
  We present High Dynamic Range Neural Radiance Fields (-NeRF) to recover an HDR radiance field from a set of low dynamic range (LDR) views with different exposures. We are able to generate both novel HDR views and novel LDR views under different exposures.
  arXiv  Detail & Related papers  (2021-11-29T11:06:39Z)
• HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset [30.249052175655606]
  We introduce a coarse-to-fine deep learning framework for HDR video reconstruction. Firstly, we perform coarse alignment and pixel blending in the image space to estimate the coarse HDR video. Secondly, we conduct more sophisticated alignment and temporal fusion in the feature space of the coarse HDR video to produce better reconstruction.
  arXiv  Detail & Related papers  (2021-03-27T16:40:05Z)
• Beyond Visual Attractiveness: Physically Plausible Single Image HDR Reconstruction for Spherical Panoramas [60.24132321381606]
  We introduce the physical illuminance constraints to our single-shot HDR reconstruction framework. Our method can generate HDRs which are not only visually appealing but also physically plausible.
  arXiv  Detail & Related papers  (2021-03-24T01:51:19Z)
• HDR-GAN: HDR Image Reconstruction from Multi-Exposed LDR Images with Large Motions [62.44802076971331]
  We propose a novel GAN-based model, HDR-GAN, for synthesizing HDR images from multi-exposed LDR images. By incorporating adversarial learning, our method is able to produce faithful information in the regions with missing content.
  arXiv  Detail & Related papers  (2020-07-03T11:42:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.