UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields

• URL: http://arxiv.org/abs/2506.21884v2
• Date: Wed, 06 Aug 2025 07:17:05 GMT
• Title: UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields
• Authors: Fabian Perez, Sara Rojas, Carlos Hinojosa, Hoover Rueda-Chacón, Bernard Ghanem,
• Abstract summary: UnMix-NeRF is a framework that integrates spectral unmixing into Neural Radiance Field (NeRF)<n>Our method models spectral reflectance via diffuse and specular components, where a learned dictionary of global endmembers represents pure material signatures.<n>For material segmentation, we use spectral signature predictions along learned endmembers, allowing unsupervised material clustering.
• Score: 44.44788548423704
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Radiance Field (NeRF)-based segmentation methods focus on object semantics and rely solely on RGB data, lacking intrinsic material properties. This limitation restricts accurate material perception, which is crucial for robotics, augmented reality, simulation, and other applications. We introduce UnMix-NeRF, a framework that integrates spectral unmixing into NeRF, enabling joint hyperspectral novel view synthesis and unsupervised material segmentation. Our method models spectral reflectance via diffuse and specular components, where a learned dictionary of global endmembers represents pure material signatures, and per-point abundances capture their distribution. For material segmentation, we use spectral signature predictions along learned endmembers, allowing unsupervised material clustering. Additionally, UnMix-NeRF enables scene editing by modifying learned endmember dictionaries for flexible material-based appearance manipulation. Extensive experiments validate our approach, demonstrating superior spectral reconstruction and material segmentation to existing methods. Project page: https://www.factral.co/UnMix-NeRF.

Related papers

• Frequency Domain-Based Diffusion Model for Unpaired Image Dehazing [92.61216319417208]
  We propose a novel frequency domain-based diffusion model, named ours, for fully exploiting the beneficial knowledge in unpaired clear data.<n>Inspired by the strong generative ability shown by Diffusion Models (DMs), we tackle the dehazing task from the perspective of frequency domain reconstruction.
  arXiv  Detail & Related papers  (2025-07-02T01:22:46Z)
• SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field [70.15900280156262]
  We propose an end-to-end Neural Radiance Field (NeRF)-based architecture for high-quality physically based rendering from a novel spectral perspective. SpectralNeRF is superior to recent NeRF-based methods when synthesizing new views on synthetic and real datasets.
  arXiv  Detail & Related papers  (2023-12-14T07:19:31Z)
• MatSpectNet: Material Segmentation Network with Domain-Aware and Physically-Constrained Hyperspectral Reconstruction [13.451692195639696]
  MatSpectNet is a new model to segment materials with recovered hyperspectral images from RGB images. It exploits the principles of colour perception in modern cameras to constrain the reconstructed hyperspectral images. It attains a 1.60% increase in average pixel accuracy and a 3.42% improvement in mean class accuracy compared with the most recent publication.
  arXiv  Detail & Related papers  (2023-07-21T10:02:02Z)
• IBL-NeRF: Image-Based Lighting Formulation of Neural Radiance Fields [12.056350920398396]
  We propose IBL-NeRF, which decomposes the neural radiance fields (NeRF) of large-scale indoor scenes into intrinsic components. Our approach inherits superior visual quality and multi-view consistency for synthesized images as well as the intrinsic components.
  arXiv  Detail & Related papers  (2022-10-15T05:38:55Z)
• NeRF-SOS: Any-View Self-supervised Object Segmentation from Complex Real-World Scenes [80.59831861186227]
  This paper carries out the exploration of self-supervised learning for object segmentation using NeRF for complex real-world scenes. Our framework, called NeRF with Self-supervised Object NeRF-SOS, encourages NeRF models to distill compact geometry-aware segmentation clusters. It consistently surpasses other 2D-based self-supervised baselines and predicts finer semantics masks than existing supervised counterparts.
  arXiv  Detail & Related papers  (2022-09-19T06:03:17Z)
• NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields [54.27264716713327]
  We show that a Neural Radiance Fields (NeRF) representation of a scene can be used to train dense object descriptors. We use an optimized NeRF to extract dense correspondences between multiple views of an object, and then use these correspondences as training data for learning a view-invariant representation of the object. Dense correspondence models supervised with our method significantly outperform off-the-shelf learned descriptors by 106%.
  arXiv  Detail & Related papers  (2022-03-03T18:49:57Z)
• NeRF in detail: Learning to sample for view synthesis [104.75126790300735]
  Neural radiance fields (NeRF) methods have demonstrated impressive novel view synthesis. In this work we address a clear limitation of the vanilla coarse-to-fine approach -- that it is based on a performance and not trained end-to-end for the task at hand. We introduce a differentiable module that learns to propose samples and their importance for the fine network, and consider and compare multiple alternatives for its neural architecture.
  arXiv  Detail & Related papers  (2021-06-09T17:59:10Z)
• Endmember-Guided Unmixing Network (EGU-Net): A General Deep Learning Framework for Self-Supervised Hyperspectral Unmixing [39.432539302311476]
  We develop a general deep learning approach for hyperspectral unmixing, called endmember-guided unmixing network (EGU-Net) EGU-Net is a two-stream Siamese deep network, which learns an additional network from the pure or nearly-pure endmembers to correct the weights of another unmixing network. The resulting general framework is not only limited to pixel-wise spectral unmixing but also applicable to spatial information modeling with convolutional operators for spatial-spectral unmixing.
  arXiv  Detail & Related papers  (2021-05-21T08:07:12Z)
• Constrained Nonnegative Matrix Factorization for Blind Hyperspectral Unmixing incorporating Endmember Independence [0.0]
  This paper presents a novel blind HU algorithm, referred to as Kurtosis-based Smooth Nonnegative Matrix Factorization (KbSNMF) It incorporates a novel constraint based on the statistical independence of the probability density functions of endmember spectra. It exhibits superior performance especially in terms of extracting endmember spectra from hyperspectral data.
  arXiv  Detail & Related papers  (2020-03-02T17:20:04Z)
• Blind Source Separation for NMR Spectra with Negative Intensity [0.0]
  We benchmark several blind source separation techniques for analysis of NMR spectral datasets containing negative intensity. FastICA, SIMPLISMA, and NNMF are top-performing techniques. The accuracy of FastICA and SIMPLISMA degrades quickly if excess (unreal) pure components are predicted.
  arXiv  Detail & Related papers  (2020-02-07T20:57:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.