SEAR: Simple and Efficient Adaptation of Visual Geometric Transformers for RGB+Thermal 3D Reconstruction

• URL: http://arxiv.org/abs/2603.18774v1
• Date: Thu, 19 Mar 2026 11:27:21 GMT
• Title: SEAR: Simple and Efficient Adaptation of Visual Geometric Transformers for RGB+Thermal 3D Reconstruction
• Abstract summary: We propose SEAR, a strategy that adapts a pretrained geometry transformer to multimodal RGB-T inputs.<n>Our approach significantly outperforms state-of-the-art methods for 3D reconstruction and camera pose estimation.<n>We validate our architecture through extensive ablation studies, demonstrating how the model aligns both modalities.
• Score: 28.61594700250136
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Foundational feed-forward visual geometry models enable accurate and efficient camera pose estimation and scene reconstruction by learning strong scene priors from massive RGB datasets. However, their effectiveness drops when applied to mixed sensing modalities, such as RGB-thermal (RGB-T) images. We observe that while a visual geometry grounded transformer pretrained on RGB data generalizes well to thermal-only reconstruction, it struggles to align RGB and thermal modalities when processed jointly. To address this, we propose SEAR, a simple yet efficient fine-tuning strategy that adapts a pretrained geometry transformer to multimodal RGB-T inputs. Despite being trained on a relatively small RGB-T dataset, our approach significantly outperforms state-of-the-art methods for 3D reconstruction and camera pose estimation, achieving significant improvements over all metrics (e.g., over 29\% in AUC@30) and delivering higher detail and consistency between modalities with negligible overhead in inference time compared to the original pretrained model. Notably, SEAR enables reliable multimodal pose estimation and reconstruction even under challenging conditions, such as low lighting and dense smoke. We validate our architecture through extensive ablation studies, demonstrating how the model aligns both modalities. Additionally, we introduce a new dataset featuring RGB and thermal sequences captured at different times, viewpoints, and illumination conditions, providing a robust benchmark for future work in multimodal 3D scene reconstruction. Code and models are publicly available at https://www.github.com/Schindler-EPFL-Lab/SEAR.

Related papers

• Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark [68.49637363691325]
  This paper offers a new perspective for RGB-NIR low-light imaging by incorporating 3D-aware neural modeling.<n>A powerful model can be optimized to implicitly fuse extremely noisy RGB observations with NIR cues in 3D space, effectively recovering clean RGB images.
  arXiv  Detail & Related papers  (2026-07-31T17:59:18Z)
• Unpaired RGB-Thermal Gaussian-Splatting Using Visual Geometric Transformers [21.564312545454925]
  Multi-modal novel view synthesis (NVS) combining RGB and thermal imagery enables precise 3D scene reconstruction with visual and thermal information.<n>We introduce a framework for unpaired RGB-thermal NVS that leverages VGGT, a 3D feed-forward transformer architecture.<n>We show that our method achieves competitive performance in thermal view synthesis while maintaining RGB fidelity.
  arXiv  Detail & Related papers  (2026-06-03T22:30:58Z)
• Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction [50.5449251266956]
  This paper addresses the task of large-scale 3D scene reconstruction from long video sequences.<n>Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints.<n>We propose a novel neural global context representation that efficiently compresses and retains long-range scene information.
  arXiv  Detail & Related papers  (2026-04-09T17:59:50Z)
• AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation [50.62485061150716]
  We present AnyHand, a large-scale synthetic dataset designed to advance the state of the art in 3D hand pose estimation.<n>We show that extending the original training sets of existing baselines with AnyHand yields significant gains on multiple benchmarks.<n>We also contribute a lightweight depth fusion module that can be easily integrated into existing RGB-based models.
  arXiv  Detail & Related papers  (2026-03-26T17:58:54Z)
• EAG3R: Event-Augmented 3D Geometry Estimation for Dynamic and Extreme-Lighting Scenes [31.16499311793949]
  E3R is a novel geometry estimation framework that augments pointmap-based reconstruction with asynchronous event streams.<n>EAG3R significantly outperforms state-of-the-art RGB-only baselines across monocular depth estimation, camera pose tracking, and dynamic reconstruction tasks.
  arXiv  Detail & Related papers  (2025-11-30T08:05:28Z)
• Pseudo Depth Meets Gaussian: A Feed-forward RGB SLAM Baseline [64.42938561167402]
  We propose an online 3D reconstruction method using 3D Gaussian-based SLAM, combined with a feed-forward recurrent prediction module.<n>This approach replaces slow test-time optimization with fast network inference, significantly improving tracking speed.<n>Our method achieves performance on par with the state-of-the-art SplaTAM, while reducing tracking time by more than 90%.
  arXiv  Detail & Related papers  (2025-08-06T16:16:58Z)
• DiMeR: Disentangled Mesh Reconstruction Model [29.827345186012558]
  DiMeR is a novel geometry-texture disentangled feed-forward model with 3D supervision for sparse-view mesh reconstruction.<n>We streamline the algorithm of mesh extraction by eliminating modules with low performance/cost ratios and redesigning regularization losses with 3D supervision.<n>Extensive experiments demonstrate that DiMeR generalises across sparse-view-, single-image-, and text-to-3D tasks, consistently outperforming baselines.
  arXiv  Detail & Related papers  (2025-04-24T15:39:20Z)
• Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference [65.42565481489132]
  Humans can easily deduce the relative pose of a previously unseen object, without labeling or training, given only a single query-reference image pair.<n>We propose a novel 3D generalizable relative pose estimation method by elaborating 3D/2.5D shape perception with a 2.5D shape from RGB-D reference.<n>Our differentiable takes the 2.5D rotatable mesh textured by the RGB and the semantic maps (obtained by DINOv2 from the RGB input), then renders new RGB and semantic maps under a novel rotated view.
  arXiv  Detail & Related papers  (2024-06-26T16:01:10Z)
• UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning [34.727262809777095]
  We propose UniRGB-IR, a scalable and efficient framework for RGB-IR semantic tasks.<n>Our framework comprises three key components: a vision transformer (ViT) foundation model, a Multi-modal Feature Pool (SFI) module, and a Supplementary Feature (SFI) module.<n> Experimental results on various RGB-IR semantic tasks demonstrate that our method can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-04-26T12:21:57Z)
• MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images [57.71600854525037]
  We propose a Fuse-Describe-Match strategy for 6D pose estimation from RGB-D images. MatchU is a generic approach that fuses 2D texture and 3D geometric cues for 6D pose prediction of unseen objects.
  arXiv  Detail & Related papers  (2024-03-03T14:01:03Z)
• Attentive Multimodal Fusion for Optical and Scene Flow [24.08052492109655]
  Existing methods typically rely solely on RGB images or fuse the modalities at later stages. We propose a novel deep neural network approach named FusionRAFT, which enables early-stage information fusion between sensor modalities. Our approach exhibits improved robustness in the presence of noise and low-lighting conditions that affect the RGB images.
  arXiv  Detail & Related papers  (2023-07-28T04:36:07Z)
• A Strong Transfer Baseline for RGB-D Fusion in Vision Transformers [0.0]
  We propose a recipe for transferring pretrained ViTs in RGB-D domains for single-view 3D object recognition. We show that our adapted ViTs score up to 95.1% top-1 accuracy in Washington, achieving new state-of-the-art results in this benchmark.
  arXiv  Detail & Related papers  (2022-10-03T12:08:09Z)
• Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images [69.5662419067878]
  Grounding referring expressions in RGBD image has been an emerging field. We present a novel task of 3D visual grounding in single-view RGBD image where the referred objects are often only partially scanned due to occlusion. Our approach first fuses the language and the visual features at the bottom level to generate a heatmap that localizes the relevant regions in the RGBD image. Then our approach conducts an adaptive feature learning based on the heatmap and performs the object-level matching with another visio-linguistic fusion to finally ground the referred object.
  arXiv  Detail & Related papers  (2021-03-14T11:18:50Z)
• Geometric Proxies for Live RGB-D Stream Enhancement and Consolidation [7.920114031312631]
  We propose a geometric superstructure for unified real-time processing of RGB-D data. Our approach consists in generating and updating through time a single set of compact local statistics parameterized over detected geometric proxies. We present experiments performed with our framework in indoor scenes of different natures within a recent open RGB-D dataset.
  arXiv  Detail & Related papers  (2020-01-21T14:42:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.