SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field

• URL: http://arxiv.org/abs/2312.08692v1
• Date: Thu, 14 Dec 2023 07:19:31 GMT
• Title: SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field
• Authors: Ru Li, Jia Liu, Guanghui Liu, Shengping Zhang, Bing Zeng, Shuaicheng Liu
• Abstract summary: We propose an end-to-end Neural Radiance Field (NeRF)-based architecture for high-quality physically based rendering from a novel spectral perspective. SpectralNeRF is superior to recent NeRF-based methods when synthesizing new views on synthetic and real datasets.
• Score: 70.15900280156262
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose SpectralNeRF, an end-to-end Neural Radiance Field (NeRF)-based architecture for high-quality physically based rendering from a novel spectral perspective. We modify the classical spectral rendering into two main steps, 1) the generation of a series of spectrum maps spanning different wavelengths, 2) the combination of these spectrum maps for the RGB output. Our SpectralNeRF follows these two steps through the proposed multi-layer perceptron (MLP)-based architecture (SpectralMLP) and Spectrum Attention UNet (SAUNet). Given the ray origin and the ray direction, the SpectralMLP constructs the spectral radiance field to obtain spectrum maps of novel views, which are then sent to the SAUNet to produce RGB images of white-light illumination. Applying NeRF to build up the spectral rendering is a more physically-based way from the perspective of ray-tracing. Further, the spectral radiance fields decompose difficult scenes and improve the performance of NeRF-based methods. Comprehensive experimental results demonstrate the proposed SpectralNeRF is superior to recent NeRF-based methods when synthesizing new views on synthetic and real datasets. The codes and datasets are available at https://github.com/liru0126/SpectralNeRF.

Related papers

• Spectral Graph Reasoning Network for Hyperspectral Image Classification [0.43512163406551996]
  Convolutional neural networks (CNNs) have achieved remarkable performance in hyperspectral image (HSI) classification. We propose a spectral graph reasoning network (SGR) learning framework comprising two crucial modules. Experiments on two HSI datasets demonstrate that the proposed architecture can significantly improve the classification accuracy.
  arXiv  Detail & Related papers  (2024-07-02T20:29:23Z)
• Spectral-wise Implicit Neural Representation for Hyperspectral Image Reconstruction [14.621504062838731]
  Coded Aperture Snapshot Spectral Imaging (CASSI) reconstruction aims to recover the 3D spatial-spectral signal from 2D measurement. Existing methods for reconstructing HSI typically involve learning mappings from a 2D compressed image to a predetermined set of discrete spectral bands. We propose an innovative method called Spectral-wise Implicit Neural Representation (SINR) as a pioneering step toward addressing this limitation.
  arXiv  Detail & Related papers  (2023-12-02T08:06:07Z)
• Spectrum-driven Mixed-frequency Network for Hyperspectral Salient Object Detection [14.621504062838731]
  We propose a novel approach that fully leverages the spectral characteristics by extracting two distinct frequency components from the spectrum. The Spectral Saliency approximates the region of salient objects, while the Spectral Edge captures edge information of salient objects. To effectively utilize this dual-frequency information, we introduce a novel lightweight Spectrum-driven Mixed-frequency Network (SMN)
  arXiv  Detail & Related papers  (2023-12-02T08:05:45Z)
• SpectralGPT: Spectral Remote Sensing Foundation Model [60.023956954916414]
  A universal RS foundation model, named SpectralGPT, is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT) Compared to existing foundation models, SpectralGPT accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience.
  arXiv  Detail & Related papers  (2023-11-13T07:09:30Z)
• Spec-NeRF: Multi-spectral Neural Radiance Fields [9.242830798112855]
  We propose Multi-spectral Neural Radiance Fields(Spec-NeRF) for jointly reconstructing a multispectral radiance field and spectral sensitivity functions(SSFs) of the camera from a set of color images filtered by different filters. Our experiments on both synthetic and real scenario datasets demonstrate that utilizing filtered RGB images with learnable NeRF and SSFs can achieve high fidelity and promising spectral reconstruction.
  arXiv  Detail & Related papers  (2023-09-14T16:17:55Z)
• Specformer: Spectral Graph Neural Networks Meet Transformers [51.644312964537356]
  Spectral graph neural networks (GNNs) learn graph representations via spectral-domain graph convolutions. We introduce Specformer, which effectively encodes the set of all eigenvalues and performs self-attention in the spectral domain. By stacking multiple Specformer layers, one can build a powerful spectral GNN.
  arXiv  Detail & Related papers  (2023-03-02T07:36:23Z)
• Cross-Spectral Neural Radiance Fields [49.28588927121722]
  We propose X-NeRF, a novel method to learn a Cross-Spectral scene representation given images captured from cameras with different light spectrum sensitivity. X-NeRF exploits Normalized Cross-Device Coordinates (NXDC) to render images of different modalities from arbitrary viewpoints.
  arXiv  Detail & Related papers  (2022-09-01T17:59:58Z)
• Continuous Spectral Reconstruction from RGB Images via Implicit Neural Representation [43.622087181097164]
  Existing methods for spectral reconstruction usually learn a discrete mapping from RGB images to a number of spectral bands. We propose Neural Spectral Reconstruction (NeSR) to lift this limitation, by introducing a novel continuous spectral representation. NeSR extends the flexibility of spectral reconstruction by enabling an arbitrary number of spectral bands as the target output.
  arXiv  Detail & Related papers  (2021-12-24T09:08:23Z)
• NeRF-SR: High-Quality Neural Radiance Fields using Super-Sampling [82.99453001445478]
  We present NeRF-SR, a solution for high-resolution (HR) novel view synthesis with mostly low-resolution (LR) inputs. Our method is built upon Neural Radiance Fields (NeRF) that predicts per-point density and color with a multi-layer perceptron.
  arXiv  Detail & Related papers  (2021-12-03T07:33:47Z)
• Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB [84.1657998542458]
  It has been proven that the reconstruction accuracy relies heavily on the spectral response of the RGB camera in use. This paper explores the filter-array based color imaging mechanism of existing RGB cameras, and proposes to design the IR-cut filter properly for improved spectral recovery.
  arXiv  Detail & Related papers  (2021-03-26T19:42:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.