Unpaired RGB-Thermal Gaussian-Splatting Using Visual Geometric Transformers

• URL: http://arxiv.org/abs/2606.05491v1
• Date: Wed, 03 Jun 2026 22:30:58 GMT
• Title: Unpaired RGB-Thermal Gaussian-Splatting Using Visual Geometric Transformers
• Abstract summary: Multi-modal novel view synthesis (NVS) combining RGB and thermal imagery enables precise 3D scene reconstruction with visual and thermal information.<n>We introduce a framework for unpaired RGB-thermal NVS that leverages VGGT, a 3D feed-forward transformer architecture.<n>We show that our method achieves competitive performance in thermal view synthesis while maintaining RGB fidelity.
• Score: 21.564312545454925
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal novel view synthesis (NVS) combining RGB and thermal imagery enables precise 3D scene reconstruction with visual and thermal information. However, existing methods typically rely on precisely calibrated RGB-thermal image pairs or stereo setups, limiting scalability and practical deployment. To address this, we introduce a framework for unpaired RGB-thermal NVS that leverages VGGT, a 3D feed-forward transformer architecture, to independently estimate camera poses for each modality. The pose sets are then aligned using the Procrustes algorithm with a cross-modal feature matcher, enabling joint registration without paired calibration. Building on this alignment, we further propose a multi-modal 3D Gaussian Splatting approach that learns directly from unpaired RGB and thermal images. Experiments on diverse scenes demonstrate that our method achieves competitive performance in thermal view synthesis while maintaining RGB fidelity. Moreover, we show that existing reconstruction approaches can produce modality-specific reconstructions that lack cross-modal consistency. We thus introduce a benchmarking framework to rigorously evaluate both per-modality image synthesis and the multi-modal coherence of reconstructed scenes.

Related papers

• Learning Spectral and Polarimetric Clues for One-to-Multimodal Novel View Synthesis [15.168593370409177]
  We propose Spectral and Polarimetric Implicit Learned Representation (SPoILeR), a novel method to obtain consistent renderings of unconventional modalities.<n>Thanks to a multimodal pre-training phase, the model learns the mutual correlation between different modalities.<n> Experimental results show that the approach can accurately render infrared, polarimetric, and multispectral frames for scenes where no input sample captured by these types of sensors is provided.
  arXiv  Detail & Related papers  (2026-07-02T16:13:19Z)
• SEAR: Simple and Efficient Adaptation of Visual Geometric Transformers for RGB+Thermal 3D Reconstruction [28.61594700250136]
  We propose SEAR, a strategy that adapts a pretrained geometry transformer to multimodal RGB-T inputs.<n>Our approach significantly outperforms state-of-the-art methods for 3D reconstruction and camera pose estimation.<n>We validate our architecture through extensive ablation studies, demonstrating how the model aligns both modalities.
  arXiv  Detail & Related papers  (2026-03-19T11:27:21Z)
• ThermoSplat: Cross-Modal 3D Gaussian Splatting with Feature Modulation and Geometry Decoupling [11.169420448510095]
  ThermoSplat is a novel framework that enables deep spectral-aware reconstruction through active feature modulation and adaptive geometry decoupling.<n>Experiments on the RGBT-Scenes dataset demonstrate that ThermoSplat achieves state-of-the-art rendering quality across both visible and thermal spectrums.
  arXiv  Detail & Related papers  (2026-01-22T12:24:26Z)
• From Restoration to Reconstruction: Rethinking 3D Gaussian Splatting for Underwater Scenes [13.730810237133822]
  We propose textbfR-Splatting, a unified framework that bridges underwater image restoration (UIR) with 3D Gaussian Splatting (3DGS)<n>Our method integrates multiple enhanced views produced by diverse UIR models into a single reconstruction pipeline.<n>Experiments on Seathru-NeRF and our new BlueCoral3D dataset demonstrate that R-Splatting outperforms strong baselines in both rendering quality and geometric accuracy.
  arXiv  Detail & Related papers  (2025-09-22T13:50:20Z)
• Towards Integrating Multi-Spectral Imaging with Gaussian Splatting [8.908394706712395]
  We present a study of how to integrate color (RGB) and multi-spectral imagery into the 3D Gaussian Splatting framework.<n>We suggest integrating multi-spectral data directly into the spherical harmonics color components to compactly model each Gaussian's multi-spectral reflectance.
  arXiv  Detail & Related papers  (2025-08-31T20:53:35Z)
• G-CUT3R: Guided 3D Reconstruction with Camera and Depth Prior Integration [57.67450930037339]
  We introduce G-CUT3R, a novel feed-forward approach for guided 3D scene reconstruction.<n>Unlike existing feed-forward methods that rely solely on input images, our method leverages auxiliary data, such as depth, camera calibrations, or camera positions.
  arXiv  Detail & Related papers  (2025-08-15T10:25:58Z)
• RGB-Phase Speckle: Cross-Scene Stereo 3D Reconstruction via Wrapped Pre-Normalization [9.20903035677888]
  This study introduces RGB-Speckle, a cross-scene 3D reconstruction framework based on an active stereo camera system.<n>We propose a novel phase pre-normalization encoding-decoding method, which mitigates external interference.<n> Experimental results demonstrate that the proposed RGB-Speckle model offers significant advantages in cross-domain and cross-scene 3D reconstruction tasks.
  arXiv  Detail & Related papers  (2025-03-08T08:37:20Z)
• GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering [83.69136534797686]
  We present GUS-IR, a novel framework designed to address the inverse rendering problem for complicated scenes featuring rough and glossy surfaces. This paper starts by analyzing and comparing two prominent shading techniques popularly used for inverse rendering, forward shading and deferred shading. We propose a unified shading solution that combines the advantages of both techniques for better decomposition.
  arXiv  Detail & Related papers  (2024-11-12T01:51:05Z)
• Breaking Modality Disparity: Harmonized Representation for Infrared and Visible Image Registration [66.33746403815283]
  We propose a scene-adaptive infrared and visible image registration. We employ homography to simulate the deformation between different planes. We propose the first ground truth available misaligned infrared and visible image dataset.
  arXiv  Detail & Related papers  (2023-04-12T06:49:56Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• SSH: A Self-Supervised Framework for Image Harmonization [97.16345684998788]
  We propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. Our results show that the proposedSSH outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study.
  arXiv  Detail & Related papers  (2021-08-15T19:51:33Z)
• Extreme Rotation Estimation using Dense Correlation Volumes [73.35119461422153]
  We present a technique for estimating the relative 3D rotation of an RGB image pair in an extreme setting. We observe that, even when images do not overlap, there may be rich hidden cues as to their geometric relationship. We propose a network design that can automatically learn such implicit cues by comparing all pairs of points between the two input images.
  arXiv  Detail & Related papers  (2021-04-28T02:00:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.