Leveraging Thermal Modality to Enhance Reconstruction in Low-Light Conditions

• URL: http://arxiv.org/abs/2403.14053v1
• Date: Thu, 21 Mar 2024 00:35:31 GMT
• Title: Leveraging Thermal Modality to Enhance Reconstruction in Low-Light Conditions
• Authors: Jiacong Xu, Mingqian Liao, K Ram Prabhakar, Vishal M. Patel,
• Abstract summary: Neural Radiance Fields (NeRF) accomplishes photo-realistic novel view synthesis by learning the implicit representation of a scene from multi-view images. Existing approaches reconstruct low-light scenes from raw images but struggle to recover texture and boundary details in dark regions. We present Thermal-NeRF, which takes thermal and visible raw images as inputs, to accomplish visible and thermal view synthesis simultaneously.
• Score: 25.14690752484963
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Fields (NeRF) accomplishes photo-realistic novel view synthesis by learning the implicit volumetric representation of a scene from multi-view images, which faithfully convey the colorimetric information. However, sensor noises will contaminate low-value pixel signals, and the lossy camera image signal processor will further remove near-zero intensities in extremely dark situations, deteriorating the synthesis performance. Existing approaches reconstruct low-light scenes from raw images but struggle to recover texture and boundary details in dark regions. Additionally, they are unsuitable for high-speed models relying on explicit representations. To address these issues, we present Thermal-NeRF, which takes thermal and visible raw images as inputs, considering the thermal camera is robust to the illumination variation and raw images preserve any possible clues in the dark, to accomplish visible and thermal view synthesis simultaneously. Also, the first multi-view thermal and visible dataset (MVTV) is established to support the research on multimodal NeRF. Thermal-NeRF achieves the best trade-off between detail preservation and noise smoothing and provides better synthesis performance than previous work. Finally, we demonstrate that both modalities are beneficial to each other in 3D reconstruction.

Related papers

• ThermalNeRF: Thermal Radiance Fields [32.881758519242155]
  We propose a unified framework for scene reconstruction from a set of LWIR and RGB images. We calibrate the RGB and infrared cameras with respect to each other, as a preprocessing step. We show that our method is capable of thermal super-resolution, as well as visually removing obstacles to reveal objects occluded in either the RGB or thermal channels.
  arXiv  Detail & Related papers  (2024-07-22T02:51:29Z)
• NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections [57.63028964831785]
  Recent works have improved NeRF's ability to render detailed specular appearance of distant environment illumination, but are unable to synthesize consistent reflections of closer content. We address these issues with an approach based on ray tracing. Instead of querying an expensive neural network for the outgoing view-dependent radiance at points along each camera ray, our model casts rays from these points and traces them through the NeRF representation to render feature vectors.
  arXiv  Detail & Related papers  (2024-05-23T17:59:57Z)
• Taming Latent Diffusion Model for Neural Radiance Field Inpainting [63.297262813285265]
  Neural Radiance Field (NeRF) is a representation for 3D reconstruction from multi-view images. We propose tempering the diffusion model'sity with per-scene customization and mitigating the textural shift with masked training. Our framework yields state-of-the-art NeRF inpainting results on various real-world scenes.
  arXiv  Detail & Related papers  (2024-04-15T17:59:57Z)
• ThermoNeRF: Multimodal Neural Radiance Fields for Thermal Novel View Synthesis [5.66229031510643]
  We propose ThermoNeRF, a novel approach to rendering new RGB and thermal views of a scene jointly. To overcome the lack of texture in thermal images, we use paired RGB and thermal images to learn scene density. We also introduce ThermoScenes, a new dataset to palliate the lack of available RGB+thermal datasets for scene reconstruction.
  arXiv  Detail & Related papers  (2024-03-18T18:10:34Z)
• NeRF-VPT: Learning Novel View Representations with Neural Radiance Fields via View Prompt Tuning [63.39461847093663]
  We propose NeRF-VPT, an innovative method for novel view synthesis to address these challenges. Our proposed NeRF-VPT employs a cascading view prompt tuning paradigm, wherein RGB information gained from preceding rendering outcomes serves as instructive visual prompts for subsequent rendering stages. NeRF-VPT only requires sampling RGB data from previous stage renderings as priors at each training stage, without relying on extra guidance or complex techniques.
  arXiv  Detail & Related papers  (2024-03-02T22:08:10Z)
• Enhancing Low-Light Images Using Infrared-Encoded Images [81.8710581927427]
  Previous arts mainly focus on the low-light images captured in the visible spectrum using pixel-wise loss. We propose a novel approach to increase the visibility of images captured under low-light environments by removing the in-camera infrared (IR) cut-off filter.
  arXiv  Detail & Related papers  (2023-07-09T08:29:19Z)
• Dual Degradation Representation for Joint Deraining and Low-Light Enhancement in the Dark [57.85378202032541]
  Rain in the dark poses a significant challenge to deploying real-world applications such as autonomous driving, surveillance systems, and night photography. Existing low-light enhancement or deraining methods struggle to brighten low-light conditions and remove rain simultaneously. We introduce an end-to-end model called L$2$RIRNet, designed to manage both low-light enhancement and deraining in real-world settings.
  arXiv  Detail & Related papers  (2023-05-06T10:17:42Z)
• Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition [78.50328335703914]
  Diffusion in the Dark (DiD) is a diffusion model for low-light image reconstruction for text recognition. We demonstrate that DiD, without any task-specific optimization, can outperform SOTA low-light methods in low-light text recognition on real images.
  arXiv  Detail & Related papers  (2023-03-07T23:52:51Z)
• BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields [9.744593647024253]
  We present a novel bundle adjusted deblur Neural Radiance Fields (BAD-NeRF) BAD-NeRF can be robust to severe motion blurred images and inaccurate camera poses. Our approach models the physical image formation process of a motion blurred image, and jointly learns the parameters of NeRF.
  arXiv  Detail & Related papers  (2022-11-23T10:53:37Z)
• NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images [37.917974033687464]
  NeRF is a technique for high quality novel view synthesis from a collection of posed input images. We modify NeRF to instead train directly on linear raw images, preserving the scene's full dynamic range. We show that NeRF is highly robust to the zero-mean distribution of raw noise.
  arXiv  Detail & Related papers  (2021-11-26T18:59:47Z)
• Thermal Image Super-Resolution Using Second-Order Channel Attention with Varying Receptive Fields [4.991042925292453]
  We introduce a system to efficiently reconstruct thermal images. The restoration of thermal images is critical for applications that involve safety, search and rescue, and military operations.
  arXiv  Detail & Related papers  (2021-07-30T22:17:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.