Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception

• URL: http://arxiv.org/abs/2605.21882v1
• Date: Thu, 21 May 2026 01:43:05 GMT
• Title: Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception
• Abstract summary: Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery.<n>We present Thermo-VL, a wavelength-aware VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module.<n>Experiments show strong gains on challenging thermal-only and RGB+thermal reasoning tasks, highlighting the value of prompt-conditioned multispectral fusion.
• Score: 45.307537865864994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery, whereas thermal infrared preserves complementary scene structure when visible cues degrade. We present Thermo-VL, a wavelength-aware VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module. Given aligned RGB tokens, thermal tokens, and prompt embeddings, the fusion module conditions thermal features on both language and RGB context, then injects a gated residual into the frozen RGB stream so thermal evidence can be incorporated without disrupting Molmo's pretrained RGB-language interface. We train the model with the standard language-modeling objective together with auxiliary alignment and regularization losses that improve cross-modal grounding and reduce over-reliance on RGB. We also introduce a pixel-aligned RGB-thermal instruction-tuning dataset and Thermo-VL-Bench, a manually screened RGB-thermal VQA benchmark for low-light and cross-spectrum reasoning. Experiments show strong gains on challenging thermal-only and RGB+thermal reasoning tasks, highlighting the value of prompt-conditioned multispectral fusion. Our dataset and code are publicly available at: https://thusharakart.github.io/Thermo-VL

Related papers

• Text2Thermal: Physics-Aware Thermal Image Synthesis from Textual Priors [7.213400161126316]
  Text2Thermal is a framework for physics-aware thermal image synthesis from textual priors.<n>Text2Thermal synthesizes thermal imagery without requiring a registered RGB image at inference.<n>On M3FD and FLIR, Text2Thermal achieves state-of-the-art FID among thermal image synthesis methods.
  arXiv  Detail & Related papers  (2026-09-03T09:33:09Z)
• Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark [68.49637363691325]
  This paper offers a new perspective for RGB-NIR low-light imaging by incorporating 3D-aware neural modeling.<n>A powerful model can be optimized to implicitly fuse extremely noisy RGB observations with NIR cues in 3D space, effectively recovering clean RGB images.
  arXiv  Detail & Related papers  (2026-07-31T17:59:18Z)
• FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models [4.917364215480415]
  FusionRS is the first large-scale RGB-infrared-text dataset for vision-language learning in remote sensing.<n>It is constructed by translating diverse public RGB remote sensing images into infrared-style counterparts, forming aligned RGB-IR image pairs.<n>Experiments show that FusionRS improves RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning over RGB-only and non-IR-aware training settings.
  arXiv  Detail & Related papers  (2026-06-15T17:49:34Z)
• Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detection [41.947503656624285]
  Existing open-vocabulary detectors focus on RGB images and fail to generalize to thermal imagery.<n>We present Thermal-Det, the first large language model (LLM) supervised open-vocabulary detector tailored for thermal images.
  arXiv  Detail & Related papers  (2026-05-11T07:41:37Z)
• TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation [12.591408054941027]
  TherA is a controllable RGB-to-TIR translation framework that produces diverse and thermally plausible images at both scene and object level.<n>TherA achieves state-of-the-art translation performance, demonstrating improved zero-shot translation performance up to 33% increase averaged across all metrics.
  arXiv  Detail & Related papers  (2026-02-23T01:56:29Z)
• ThermalGen: Style-Disentangled Flow-Based Generative Models for RGB-to-Thermal Image Translation [14.108149959967095]
  Paired RGB-thermal data is crucial for visual-thermal sensor fusion and cross-modality tasks.<n>To overcome this challenge, RGB-to-Thermal (RGB-T) image translation has emerged as a promising solution.<n>We propose ThermalGen, an adaptive flow-based generative model for RGB-T image translation.
  arXiv  Detail & Related papers  (2025-09-29T14:55:51Z)
• Human Activity Recognition using RGB-Event based Sensors: A Multi-modal Heat Conduction Model and A Benchmark Dataset [65.76480665062363]
  Human Activity Recognition primarily relied on traditional RGB cameras to achieve high-performance activity recognition.<n>Challenges in real-world scenarios, such as insufficient lighting and rapid movements, inevitably degrade the performance of RGB cameras.<n>In this work, we rethink human activity recognition by combining the RGB and event cameras.
  arXiv  Detail & Related papers  (2025-04-08T09:14:24Z)
• Bringing RGB and IR Together: Hierarchical Multi-Modal Enhancement for Robust Transmission Line Detection [67.02804741856512]
  We propose a novel Hierarchical Multi-Modal Enhancement Network (HMMEN) that integrates RGB and IR data for robust and accurate TL detection.<n>Our method introduces two key components: (1) a Mutual Multi-Modal Enhanced Block (MMEB), which fuses and enhances hierarchical RGB and IR feature maps in a coarse-to-fine manner, and (2) a Feature Alignment Block (FAB) that corrects misalignments between decoder outputs and IR feature maps by leveraging deformable convolutions.
  arXiv  Detail & Related papers  (2025-01-25T06:21:06Z)
• Does Thermal Really Always Matter for RGB-T Salient Object Detection? [153.17156598262656]
  This paper proposes a network named TNet to solve the RGB-T salient object detection (SOD) task. In this paper, we introduce a global illumination estimation module to predict the global illuminance score of the image. On the other hand, we introduce a two-stage localization and complementation module in the decoding phase to transfer object localization cue and internal integrity cue in thermal features to the RGB modality.
  arXiv  Detail & Related papers  (2022-10-09T13:50:12Z)
• Mirror Complementary Transformer Network for RGB-thermal Salient Object Detection [16.64781797503128]
  RGB-thermal object detection (RGB-T SOD) aims to locate the common prominent objects of an aligned visible and thermal infrared image pair. In this paper, we propose a novel mirror complementary Transformer network (MCNet) for RGB-T SOD. Experiments on benchmark and VT723 datasets show that the proposed method outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-07-07T20:26:09Z)
• Dual Swin-Transformer based Mutual Interactive Network for RGB-D Salient Object Detection [67.33924278729903]
  In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the long-range dependencies in visual inputs. Comprehensive experiments on five standard RGB-D SOD benchmark datasets demonstrate the superiority of the proposed DTMINet method.
  arXiv  Detail & Related papers  (2022-06-07T08:35:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.