Hyperspectral Neural Radiance Fields

• URL: http://arxiv.org/abs/2403.14839v1
• Date: Thu, 21 Mar 2024 21:18:08 GMT
• Title: Hyperspectral Neural Radiance Fields
• Authors: Gerry Chen, Sunil Kumar Narayanan, Thomas Gautier Ottou, Benjamin Missaoui, Harsh Muriki, Cédric Pradalier, Yongsheng Chen,
• Abstract summary: We propose a hyperspectral 3D reconstruction using Neural Radiance Fields (NeRFs) NeRFs have seen widespread success in creating high quality volumetric 3D representations of scenes captured by a variety of camera models. We show that our hyperspectral NeRF approach enables creating fast, accurate volumetric 3D hyperspectral scenes.
• Score: 11.485829401765521
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hyperspectral Imagery (HSI) has been used in many applications to non-destructively determine the material and/or chemical compositions of samples. There is growing interest in creating 3D hyperspectral reconstructions, which could provide both spatial and spectral information while also mitigating common HSI challenges such as non-Lambertian surfaces and translucent objects. However, traditional 3D reconstruction with HSI is difficult due to technological limitations of hyperspectral cameras. In recent years, Neural Radiance Fields (NeRFs) have seen widespread success in creating high quality volumetric 3D representations of scenes captured by a variety of camera models. Leveraging recent advances in NeRFs, we propose computing a hyperspectral 3D reconstruction in which every point in space and view direction is characterized by wavelength-dependent radiance and transmittance spectra. To evaluate our approach, a dataset containing nearly 2000 hyperspectral images across 8 scenes and 2 cameras was collected. We perform comparisons against traditional RGB NeRF baselines and apply ablation testing with alternative spectra representations. Finally, we demonstrate the potential of hyperspectral NeRFs for hyperspectral super-resolution and imaging sensor simulation. We show that our hyperspectral NeRF approach enables creating fast, accurate volumetric 3D hyperspectral scenes and enables several new applications and areas for future study.

Related papers

• Beyond the Visible: Jointly Attending to Spectral and Spatial Dimensions with HSI-Diffusion for the FINCH Spacecraft [2.5057561650768814]
  The FINCH mission aims to monitor crop residue cover in agricultural fields. Hyperspectral imaging captures both spectral and spatial information. It is prone to various types of noise, including random noise, stripe noise, and dead pixels.
  arXiv  Detail & Related papers  (2024-06-15T19:34:18Z)
• Multispectral Stereo-Image Fusion for 3D Hyperspectral Scene Reconstruction [4.2056926734482065]
  We present a novel approach combining two calibrated multispectral real-time capable snapshot cameras, covering different spectral ranges, into a stereo-system. The combined use of different multispectral snapshot cameras enables both 3D reconstruction and spectral analysis.
  arXiv  Detail & Related papers  (2023-12-15T13:20:35Z)
• StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D [88.66678730537777]
  We present StableDreamer, a methodology incorporating three advances. First, we formalize the equivalence of the SDS generative prior and a simple supervised L2 reconstruction loss. Second, our analysis shows that while image-space diffusion contributes to geometric precision, latent-space diffusion is crucial for vivid color rendition.
  arXiv  Detail & Related papers  (2023-12-02T02:27:58Z)
• Spec-NeRF: Multi-spectral Neural Radiance Fields [9.242830798112855]
  We propose Multi-spectral Neural Radiance Fields(Spec-NeRF) for jointly reconstructing a multispectral radiance field and spectral sensitivity functions(SSFs) of the camera from a set of color images filtered by different filters. Our experiments on both synthetic and real scenario datasets demonstrate that utilizing filtered RGB images with learnable NeRF and SSFs can achieve high fidelity and promising spectral reconstruction.
  arXiv  Detail & Related papers  (2023-09-14T16:17:55Z)
• Multi-Space Neural Radiance Fields [74.46513422075438]
  Existing Neural Radiance Fields (NeRF) methods suffer from the existence of reflective objects. We propose a multi-space neural radiance field (MS-NeRF) that represents the scene using a group of feature fields in parallel sub-spaces. Our approach significantly outperforms the existing single-space NeRF methods for rendering high-quality scenes.
  arXiv  Detail & Related papers  (2023-05-07T13:11:07Z)
• Clean-NeRF: Reformulating NeRF to account for View-Dependent Observations [67.54358911994967]
  This paper proposes Clean-NeRF for accurate 3D reconstruction and novel view rendering in complex scenes. Clean-NeRF can be implemented as a plug-in that can immediately benefit existing NeRF-based methods without additional input.
  arXiv  Detail & Related papers  (2023-03-26T12:24:31Z)
• NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes [56.31855837632735]
  We propose a compact and flexible architecture that enables easy 3D surface reconstruction from any NeRF-driven approach. Our final 3D mesh is physically accurate and can be rendered in real time on an array of devices.
  arXiv  Detail & Related papers  (2023-03-16T16:06:03Z)
• NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion [107.67277084886929]
  Novel view synthesis from a single image requires inferring occluded regions of objects and scenes whilst simultaneously maintaining semantic and physical consistency with the input. We propose NerfDiff, which addresses this issue by distilling the knowledge of a 3D-aware conditional diffusion model (CDM) into NeRF through synthesizing and refining a set of virtual views at test time. We further propose a novel NeRF-guided distillation algorithm that simultaneously generates 3D consistent virtual views from the CDM samples, and finetunes the NeRF based on the improved virtual views.
  arXiv  Detail & Related papers  (2023-02-20T17:12:00Z)
• HyperNeRFGAN: Hypernetwork approach to 3D NeRF GAN [0.4606926019499777]
  This paper introduces HyperNeRFGAN, a Generative Adversarial Network (GAN) architecture employing a hypernetwork paradigm to transform a Gaussian noise into the weights of a NeRF architecture. Despite its notable simplicity in comparison to existing state-of-the-art alternatives, the proposed model demonstrates superior performance on a diverse range of image datasets.
  arXiv  Detail & Related papers  (2023-01-27T10:21:18Z)
• Fast Hyperspectral Image Recovery via Non-iterative Fusion of Dual-Camera Compressive Hyperspectral Imaging [22.683482662362337]
  Coded aperture snapshot spectral imaging (CASSI) is a promising technique to capture the three-dimensional hyperspectral image (HSI) Various regularizers have been exploited to reconstruct the 3D data from the 2D measurement. One feasible solution is to utilize additional information such as the RGB measurement in CASSI.
  arXiv  Detail & Related papers  (2020-12-30T10:29:32Z)
• Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery [79.69449412334188]
  In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches. We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
  arXiv  Detail & Related papers  (2020-05-18T14:25:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.