Towards Integrating Multi-Spectral Imaging with Gaussian Splatting

• URL: http://arxiv.org/abs/2509.00989v1
• Date: Sun, 31 Aug 2025 20:53:35 GMT
• Title: Towards Integrating Multi-Spectral Imaging with Gaussian Splatting
• Abstract summary: We present a study of how to integrate color (RGB) and multi-spectral imagery into the 3D Gaussian Splatting framework.<n>We suggest integrating multi-spectral data directly into the spherical harmonics color components to compactly model each Gaussian's multi-spectral reflectance.
• Score: 8.908394706712395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a study of how to integrate color (RGB) and multi-spectral imagery (red, green, red-edge, and near-infrared) into the 3D Gaussian Splatting (3DGS) framework, a state-of-the-art explicit radiance-field-based method for fast and high-fidelity 3D reconstruction from multi-view images. While 3DGS excels on RGB data, naive per-band optimization of additional spectra yields poor reconstructions due to inconsistently appearing geometry in the spectral domain. This problem is prominent, even though the actual geometry is the same, regardless of spectral modality. To investigate this, we evaluate three strategies: 1) Separate per-band reconstruction with no shared structure. 2) Splitting optimization, in which we first optimize RGB geometry, copy it, and then fit each new band to the model by optimizing both geometry and band representation. 3) Joint, in which the modalities are jointly optimized, optionally with an initial RGB-only phase. We showcase through quantitative metrics and qualitative novel-view renderings on multi-spectral datasets the effectiveness of our dedicated optimized Joint strategy, increasing overall spectral reconstruction as well as enhancing RGB results through spectral cross-talk. We therefore suggest integrating multi-spectral data directly into the spherical harmonics color components to compactly model each Gaussian's multi-spectral reflectance. Moreover, our analysis reveals several key trade-offs in when and how to introduce spectral bands during optimization, offering practical insights for robust multi-modal 3DGS reconstruction.

Related papers

• Unpaired RGB-Thermal Gaussian-Splatting Using Visual Geometric Transformers [21.564312545454925]
  Multi-modal novel view synthesis (NVS) combining RGB and thermal imagery enables precise 3D scene reconstruction with visual and thermal information.<n>We introduce a framework for unpaired RGB-thermal NVS that leverages VGGT, a 3D feed-forward transformer architecture.<n>We show that our method achieves competitive performance in thermal view synthesis while maintaining RGB fidelity.
  arXiv  Detail & Related papers  (2026-06-03T22:30:58Z)
• Residual Gaussian Splatting for Ultra Sparse-View CBCT Reconstruction [14.00351191217297]
  This work integrates wavelet multi-resolution analysis with 3DGS to enhance sparse reconstruction.<n>We introduce a spectrally-decoupled Gaussian representation that stratifies the volumetric field into a geometric base component and a residual detail component.<n>Experiments on clinical datasets demonstrate that RGS enables the reconstructed images to capture highly refined geometric textures.
  arXiv  Detail & Related papers  (2026-04-30T07:59:18Z)
• MSGS: Multispectral 3D Gaussian Splatting [46.61006851229471]
  We present a multispectral extension to 3D Gaussian Splatting (3DGS) for wavelength-aware view synthesis.<n>Each Gaussian is augmented with spectral radiance, represented via per-band spherical harmonics, and optimized under a dual-loss supervision scheme combining RGB and multispectral signals.<n>Our method is evaluated on both public and self-captured real-world datasets, demonstrating consistent improvements over the RGB-only 3DGS baseline in terms of image quality and spectral consistency.
  arXiv  Detail & Related papers  (2026-04-14T23:03:38Z)
• UniGS: Unified Geometry-Aware Gaussian Splatting for Multimodal Rendering [10.560500427919647]
  We propose UniGS, a unified map representation and differentiable attribute reconstruction based on 3D Splatting.<n>Our framework integrates a multimodal viewer capable of rendering photo-realistic RGB images, geometrically accurate depth maps, consistent surface normals, and semantic logits simultaneously.
  arXiv  Detail & Related papers  (2025-10-14T06:07:57Z)
• Multispectral-NeRF:a multispectral modeling approach based on neural radiance fields [3.606065291262699]
  3D reconstruction techniques based on 2D images typically rely on RGB spectral information.<n>Additional spectral bands beyond RGB have been increasingly incorporated into 3D reconstruction.<n>Existing methods that integrate these expanded spectral data often suffer from expensive scheme prices, low accuracy and poor geometric features.<n>We propose Multispectral-NeRF, an enhanced neural architecture derived from NeRF that can effectively integrate multispectral information.
  arXiv  Detail & Related papers  (2025-09-14T09:04:35Z)
• Multi-Spectral Gaussian Splatting with Neural Color Representation [8.200719250787651]
  We present MS-Splatting -- a multi-spectral 3D Gaussian Splatting framework.<n>Our method does not require cross-modal camera calibration.<n>It is versatile enough to model a variety of different spectra, including thermal and near-infra red.
  arXiv  Detail & Related papers  (2025-06-03T21:36:50Z)
• Generalizable and Relightable Gaussian Splatting for Human Novel View Synthesis [49.67420486373202]
  GRGS is a generalizable and relightable 3D Gaussian framework for high-fidelity human novel view synthesis under diverse lighting conditions.<n>We introduce a Lighting-aware Geometry Refinement (LGR) module trained on synthetically relit data to predict accurate depth and surface normals.
  arXiv  Detail & Related papers  (2025-05-27T17:59:47Z)
• Unleashing Correlation and Continuity for Hyperspectral Reconstruction from RGB Images [64.80875911446937]
  We propose a Correlation and Continuity Network (CCNet) for HSI reconstruction from RGB images.<n>For the correlation of local spectrum, we introduce the Group-wise Spectral Correlation Modeling (GrSCM) module.<n>For the continuity of global spectrum, we design the Neighborhood-wise Spectral Continuity Modeling (NeSCM) module.
  arXiv  Detail & Related papers  (2025-01-02T15:14:40Z)
• GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering [83.69136534797686]
  We present GUS-IR, a novel framework designed to address the inverse rendering problem for complicated scenes featuring rough and glossy surfaces. This paper starts by analyzing and comparing two prominent shading techniques popularly used for inverse rendering, forward shading and deferred shading. We propose a unified shading solution that combines the advantages of both techniques for better decomposition.
  arXiv  Detail & Related papers  (2024-11-12T01:51:05Z)
• GeoGS3D: Single-view 3D Reconstruction via Geometric-aware Diffusion Model and Gaussian Splatting [81.03553265684184]
  We introduce GeoGS3D, a framework for reconstructing detailed 3D objects from single-view images. We propose a novel metric, Gaussian Divergence Significance (GDS), to prune unnecessary operations during optimization. Experiments demonstrate that GeoGS3D generates images with high consistency across views and reconstructs high-quality 3D objects.
  arXiv  Detail & Related papers  (2024-03-15T12:24:36Z)
• SpectralGPT: Spectral Remote Sensing Foundation Model [60.023956954916414]
  A universal RS foundation model, named SpectralGPT, is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT) Compared to existing foundation models, SpectralGPT accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience.
  arXiv  Detail & Related papers  (2023-11-13T07:09:30Z)
• Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB [84.1657998542458]
  It has been proven that the reconstruction accuracy relies heavily on the spectral response of the RGB camera in use. This paper explores the filter-array based color imaging mechanism of existing RGB cameras, and proposes to design the IR-cut filter properly for improved spectral recovery.
  arXiv  Detail & Related papers  (2021-03-26T19:42:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.