SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregation

• URL: http://arxiv.org/abs/2508.17316v1
• Date: Sun, 24 Aug 2025 11:54:16 GMT
• Title: SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregation
• Authors: Zhenyu Jin, Wenjie Li, Zhanyu Ma, Heng Guo,
• Abstract summary: SpecGen is a novel method that generates spectral bidirectional reflectance distribution functions (BRDFs) from a single RGB image of a sphere.<n>We propose the Spectral-Spatial Tri-plane Aggregation (SSTA) network, which models reflectance responses across wavelengths and incident-outgoing directions.<n>Experiments show that our method accurately reconstructs spectral BRDFs from limited spectral data and surpasses state-of-the-art methods in hyperspectral image reconstruction.
• Score: 35.544934250198374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthesizing spectral images across different wavelengths is essential for photorealistic rendering. Unlike conventional spectral uplifting methods that convert RGB images into spectral ones, we introduce SpecGen, a novel method that generates spectral bidirectional reflectance distribution functions (BRDFs) from a single RGB image of a sphere. This enables spectral image rendering under arbitrary illuminations and shapes covered by the corresponding material. A key challenge in spectral BRDF generation is the scarcity of measured spectral BRDF data. To address this, we propose the Spectral-Spatial Tri-plane Aggregation (SSTA) network, which models reflectance responses across wavelengths and incident-outgoing directions, allowing the training strategy to leverage abundant RGB BRDF data to enhance spectral BRDF generation. Experiments show that our method accurately reconstructs spectral BRDFs from limited spectral data and surpasses state-of-the-art methods in hyperspectral image reconstruction, achieving an improvement of 8 dB in PSNR. Codes and data will be released upon acceptance.

Related papers

• RGB Pre-Training Enhanced Unobservable Feature Latent Diffusion Model for Spectral Reconstruction [16.54284634377436]
  We propose a two-stage pipeline consisting of spectral structure representation learning and spectral-spatial joint distribution learning.<n>In the first stage, a spectral unobservable feature autoencoder (SpeUAE) is trained to extract and compress the unobservable feature into a 3D manifold aligned with RGB space.<n>The ULDM is then acquired to model the distribution of the coded unobservable feature with guidance from the corresponding RGB images.
  arXiv  Detail & Related papers  (2025-07-17T10:07:32Z)
• FRN: Fractal-Based Recursive Spectral Reconstruction Network [32.54705293932158]
  spectral reconstruction can significantly reduce the cost of hyperspectral images (HSIs) from RGB images.<n>We propose a Fractal-Based Recursive Spectral Reconstruction Network (FRN) which treats spectral reconstruction as a progressive process.<n> FRN achieves superior reconstruction performance compared to state-of-the-art methods in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2025-05-21T12:20:59Z)
• CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis [75.25966323298003]
  Spectral imaging offers promising applications across diverse domains, including medicine and urban scene understanding.<n> variability in channel dimensionality and captured wavelengths among spectral cameras impede the development of AI-driven methodologies.<n>We introduce $textbfCARL$, a model for $textbfC$amera-$textbfA$gnostic $textbfR$esupervised $textbfL$ across RGB, multispectral, and hyperspectral imaging modalities.
  arXiv  Detail & Related papers  (2025-04-27T13:06:40Z)
• Contourlet Refinement Gate Framework for Thermal Spectrum Distribution Regularized Infrared Image Super-Resolution [54.293362972473595]
  Image super-resolution (SR) aims to reconstruct high-resolution (HR) images from their low-resolution (LR) counterparts. Current approaches to address SR tasks are either dedicated to extracting RGB image features or assuming similar degradation patterns. We propose a Contourlet refinement gate framework to restore infrared modal-specific features while preserving spectral distribution fidelity.
  arXiv  Detail & Related papers  (2024-11-19T14:24:03Z)
• SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field [70.15900280156262]
  We propose an end-to-end Neural Radiance Field (NeRF)-based architecture for high-quality physically based rendering from a novel spectral perspective. SpectralNeRF is superior to recent NeRF-based methods when synthesizing new views on synthetic and real datasets.
  arXiv  Detail & Related papers  (2023-12-14T07:19:31Z)
• SpectralGPT: Spectral Remote Sensing Foundation Model [60.023956954916414]
  A universal RS foundation model, named SpectralGPT, is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT) Compared to existing foundation models, SpectralGPT accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience.
  arXiv  Detail & Related papers  (2023-11-13T07:09:30Z)
• Continuous Spectral Reconstruction from RGB Images via Implicit Neural Representation [43.622087181097164]
  Existing methods for spectral reconstruction usually learn a discrete mapping from RGB images to a number of spectral bands. We propose Neural Spectral Reconstruction (NeSR) to lift this limitation, by introducing a novel continuous spectral representation. NeSR extends the flexibility of spectral reconstruction by enabling an arbitrary number of spectral bands as the target output.
  arXiv  Detail & Related papers  (2021-12-24T09:08:23Z)
• Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB [84.1657998542458]
  It has been proven that the reconstruction accuracy relies heavily on the spectral response of the RGB camera in use. This paper explores the filter-array based color imaging mechanism of existing RGB cameras, and proposes to design the IR-cut filter properly for improved spectral recovery.
  arXiv  Detail & Related papers  (2021-03-26T19:42:21Z)
• Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery [79.69449412334188]
  In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches. We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
  arXiv  Detail & Related papers  (2020-05-18T14:25:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.