Learning Spectral and Polarimetric Clues for One-to-Multimodal Novel View Synthesis

• URL: http://arxiv.org/abs/2607.02372v1
• Date: Thu, 02 Jul 2026 16:13:19 GMT
• Title: Learning Spectral and Polarimetric Clues for One-to-Multimodal Novel View Synthesis
• Abstract summary: We propose Spectral and Polarimetric Implicit Learned Representation (SPoILeR), a novel method to obtain consistent renderings of unconventional modalities.<n>Thanks to a multimodal pre-training phase, the model learns the mutual correlation between different modalities.<n> Experimental results show that the approach can accurately render infrared, polarimetric, and multispectral frames for scenes where no input sample captured by these types of sensors is provided.
• Score: 15.168593370409177
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Neural rendering techniques allow for accurate reconstruction of the geometry and color appearance of 3D scenes. Some methods have extended their use to additional imaging modalities, such as multispectral, infrared, or polarimetric data. However, all of these approaches require expensive sensors and calibrated setups to capture new multimodal frames for each new scene. We propose Spectral and Polarimetric Implicit Learned Representation (SPoILeR), a novel method to obtain multi-view consistent renderings of unconventional modalities for scenes where either only RGB frames or very few of the additional modalities are available. Thanks to a multimodal pre-training phase, the model learns the mutual correlation between different modalities. This step allows predicting accurate renderings of unconventional modalities during a fine-tuning phase supervised only by RGB images. Experimental results show that the approach can accurately render infrared, polarimetric, and multispectral frames for scenes where no input sample captured by these types of sensors is provided.

Related papers

• Unpaired RGB-Thermal Gaussian-Splatting Using Visual Geometric Transformers [21.564312545454925]
  Multi-modal novel view synthesis (NVS) combining RGB and thermal imagery enables precise 3D scene reconstruction with visual and thermal information.<n>We introduce a framework for unpaired RGB-thermal NVS that leverages VGGT, a 3D feed-forward transformer architecture.<n>We show that our method achieves competitive performance in thermal view synthesis while maintaining RGB fidelity.
  arXiv  Detail & Related papers  (2026-06-03T22:30:58Z)
• Multi-Spectral Gaussian Splatting with Neural Color Representation [8.200719250787651]
  We present MS-Splatting -- a multi-spectral 3D Gaussian Splatting framework.<n>Our method does not require cross-modal camera calibration.<n>It is versatile enough to model a variety of different spectra, including thermal and near-infra red.
  arXiv  Detail & Related papers  (2025-06-03T21:36:50Z)
• DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion [53.70278210626701]
  We propose a data-driven multi-view reasoning approach that directly infers 3D scene geometry and camera poses from multi-view images.<n>Our framework, DiffusionSfM, parameterizes scene geometry and cameras as pixel-wise ray origins and endpoints in a global frame.<n>We empirically validate DiffusionSfM on both synthetic and real datasets, demonstrating that it outperforms classical and learning-based approaches.
  arXiv  Detail & Related papers  (2025-05-08T17:59:47Z)
• Multispectral Texture Synthesis using RGB Convolutional Neural Networks [2.3213238782019316]
  State-of-the-art RGB texture synthesis algorithms rely on style distances that are computed through statistics of deep features. We propose two solutions to extend these methods to multispectral imaging.
  arXiv  Detail & Related papers  (2024-10-21T13:49:54Z)
• MultiDiff: Consistent Novel View Synthesis from a Single Image [60.04215655745264]
  MultiDiff is a novel approach for consistent novel view synthesis of scenes from a single RGB image. Our results demonstrate that MultiDiff outperforms state-of-the-art methods on the challenging, real-world datasets RealEstate10K and ScanNet.
  arXiv  Detail & Related papers  (2024-06-26T17:53:51Z)
• Spec-NeRF: Multi-spectral Neural Radiance Fields [9.242830798112855]
  We propose Multi-spectral Neural Radiance Fields(Spec-NeRF) for jointly reconstructing a multispectral radiance field and spectral sensitivity functions(SSFs) of the camera from a set of color images filtered by different filters. Our experiments on both synthetic and real scenario datasets demonstrate that utilizing filtered RGB images with learnable NeRF and SSFs can achieve high fidelity and promising spectral reconstruction.
  arXiv  Detail & Related papers  (2023-09-14T16:17:55Z)
• Breaking Modality Disparity: Harmonized Representation for Infrared and Visible Image Registration [66.33746403815283]
  We propose a scene-adaptive infrared and visible image registration. We employ homography to simulate the deformation between different planes. We propose the first ground truth available misaligned infrared and visible image dataset.
  arXiv  Detail & Related papers  (2023-04-12T06:49:56Z)
• Enhancement of Novel View Synthesis Using Omnidirectional Image Completion [61.78187618370681]
  We present a method for synthesizing novel views from a single 360-degree RGB-D image based on the neural radiance field (NeRF) Experiments demonstrated that the proposed method can synthesize plausible novel views while preserving the features of the scene for both artificial and real-world data.
  arXiv  Detail & Related papers  (2022-03-18T13:49:25Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras [63.186486240525554]
  DeepMultiCap is a novel method for multi-person performance capture using sparse multi-view cameras. Our method can capture time varying surface details without the need of using pre-scanned template models.
  arXiv  Detail & Related papers  (2021-05-01T14:32:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.