Towards Integrating Multi-Spectral Imaging with Gaussian Splatting

• URL: http://arxiv.org/abs/2509.00989v1
• Date: Sun, 31 Aug 2025 20:53:35 GMT
• Title: Towards Integrating Multi-Spectral Imaging with Gaussian Splatting
• Authors: Josef Grün, Lukas Meyer, Maximilian Weiherer, Bernhard Egger, Marc Stamminger, Linus Franke,
• Abstract summary: We present a study of how to integrate color (RGB) and multi-spectral imagery into the 3D Gaussian Splatting framework.<n>We suggest integrating multi-spectral data directly into the spherical harmonics color components to compactly model each Gaussian's multi-spectral reflectance.
• Score: 8.908394706712395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a study of how to integrate color (RGB) and multi-spectral imagery (red, green, red-edge, and near-infrared) into the 3D Gaussian Splatting (3DGS) framework, a state-of-the-art explicit radiance-field-based method for fast and high-fidelity 3D reconstruction from multi-view images. While 3DGS excels on RGB data, naive per-band optimization of additional spectra yields poor reconstructions due to inconsistently appearing geometry in the spectral domain. This problem is prominent, even though the actual geometry is the same, regardless of spectral modality. To investigate this, we evaluate three strategies: 1) Separate per-band reconstruction with no shared structure. 2) Splitting optimization, in which we first optimize RGB geometry, copy it, and then fit each new band to the model by optimizing both geometry and band representation. 3) Joint, in which the modalities are jointly optimized, optionally with an initial RGB-only phase. We showcase through quantitative metrics and qualitative novel-view renderings on multi-spectral datasets the effectiveness of our dedicated optimized Joint strategy, increasing overall spectral reconstruction as well as enhancing RGB results through spectral cross-talk. We therefore suggest integrating multi-spectral data directly into the spherical harmonics color components to compactly model each Gaussian's multi-spectral reflectance. Moreover, our analysis reveals several key trade-offs in when and how to introduce spectral bands during optimization, offering practical insights for robust multi-modal 3DGS reconstruction.

Related papers

• UniGS: Unified Geometry-Aware Gaussian Splatting for Multimodal Rendering [10.560500427919647]
  We propose UniGS, a unified map representation and differentiable attribute reconstruction based on 3D Splatting.<n>Our framework integrates a multimodal viewer capable of rendering photo-realistic RGB images, geometrically accurate depth maps, consistent surface normals, and semantic logits simultaneously.
  arXiv  Detail & Related papers  (2025-10-14T06:07:57Z)
• Multispectral-NeRF:a multispectral modeling approach based on neural radiance fields [3.606065291262699]
  3D reconstruction techniques based on 2D images typically rely on RGB spectral information.<n>Additional spectral bands beyond RGB have been increasingly incorporated into 3D reconstruction.<n>Existing methods that integrate these expanded spectral data often suffer from expensive scheme prices, low accuracy and poor geometric features.<n>We propose Multispectral-NeRF, an enhanced neural architecture derived from NeRF that can effectively integrate multispectral information.
  arXiv  Detail & Related papers  (2025-09-14T09:04:35Z)
• Multi-Spectral Gaussian Splatting with Neural Color Representation [8.200719250787651]
  We present MS-Splatting -- a multi-spectral 3D Gaussian Splatting framework.<n>Our method does not require cross-modal camera calibration.<n>It is versatile enough to model a variety of different spectra, including thermal and near-infra red.
  arXiv  Detail & Related papers  (2025-06-03T21:36:50Z)
• Generalizable and Relightable Gaussian Splatting for Human Novel View Synthesis [49.67420486373202]
  GRGS is a generalizable and relightable 3D Gaussian framework for high-fidelity human novel view synthesis under diverse lighting conditions.<n>We introduce a Lighting-aware Geometry Refinement (LGR) module trained on synthetically relit data to predict accurate depth and surface normals.
  arXiv  Detail & Related papers  (2025-05-27T17:59:47Z)
• Unleashing Correlation and Continuity for Hyperspectral Reconstruction from RGB Images [64.80875911446937]
  We propose a Correlation and Continuity Network (CCNet) for HSI reconstruction from RGB images.<n>For the correlation of local spectrum, we introduce the Group-wise Spectral Correlation Modeling (GrSCM) module.<n>For the continuity of global spectrum, we design the Neighborhood-wise Spectral Continuity Modeling (NeSCM) module.
  arXiv  Detail & Related papers  (2025-01-02T15:14:40Z)
• GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering [83.69136534797686]
  We present GUS-IR, a novel framework designed to address the inverse rendering problem for complicated scenes featuring rough and glossy surfaces. This paper starts by analyzing and comparing two prominent shading techniques popularly used for inverse rendering, forward shading and deferred shading. We propose a unified shading solution that combines the advantages of both techniques for better decomposition.
  arXiv  Detail & Related papers  (2024-11-12T01:51:05Z)
• GeoGS3D: Single-view 3D Reconstruction via Geometric-aware Diffusion Model and Gaussian Splatting [81.03553265684184]
  We introduce GeoGS3D, a framework for reconstructing detailed 3D objects from single-view images. We propose a novel metric, Gaussian Divergence Significance (GDS), to prune unnecessary operations during optimization. Experiments demonstrate that GeoGS3D generates images with high consistency across views and reconstructs high-quality 3D objects.
  arXiv  Detail & Related papers  (2024-03-15T12:24:36Z)
• SpectralGPT: Spectral Remote Sensing Foundation Model [60.023956954916414]
  A universal RS foundation model, named SpectralGPT, is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT) Compared to existing foundation models, SpectralGPT accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience.
  arXiv  Detail & Related papers  (2023-11-13T07:09:30Z)
• Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB [84.1657998542458]
  It has been proven that the reconstruction accuracy relies heavily on the spectral response of the RGB camera in use. This paper explores the filter-array based color imaging mechanism of existing RGB cameras, and proposes to design the IR-cut filter properly for improved spectral recovery.
  arXiv  Detail & Related papers  (2021-03-26T19:42:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.