FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models

• URL: http://arxiv.org/abs/2606.17020v1
• Date: Mon, 15 Jun 2026 17:49:34 GMT
• Title: FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models
• Abstract summary: FusionRS is the first large-scale RGB-infrared-text dataset for vision-language learning in remote sensing.<n>It is constructed by translating diverse public RGB remote sensing images into infrared-style counterparts, forming aligned RGB-IR image pairs.<n>Experiments show that FusionRS improves RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning over RGB-only and non-IR-aware training settings.
• Score: 4.917364215480415
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Remote sensing vision-language models have advanced Earth observation understanding, but most existing work remains centered on RGB imagery, leaving the complementary information in infrared data underexplored. Infrared images provide distinctive cues, including thermal intensity structures, object boundaries, and illumination-invariant scene features, which can enrich visual-language learning beyond conventional RGB observations. However, a large-scale RGB-infrared-text dataset for remote sensing vision-language modeling is still absent. To address this gap, we introduce FusionRS, the first large-scale RGB-infrared-text dataset designed for dual-modal vision-language learning in remote sensing. FusionRS is constructed by translating diverse public RGB remote sensing images into infrared-style counterparts, forming aligned RGB-IR image pairs. Each pair is associated with conventional scene captions and IR-aware captions that explicitly describe infrared-specific visual properties while preserving semantic content. Based on FusionRS, we train dual-modal vision-language foundation models for RGB-IR joint understanding. We first train CLIP-style models for RGB-IR-text alignment, and then fine-tune generative VLMs for dual-modal RGB-IR captioning. Experiments show that FusionRS improves RGB-IR alignment, infrared-to-text retrieval, and dual-modal captioning over RGB-only and non-IR-aware training settings. Ablation studies further verify that IR-aware captions are crucial for strengthening infrared-language alignment, highlighting the importance of modality-specific textual supervision for more scalable RGB-infrared remote sensing vision-language representation learning.

Related papers

• MonoIR-RS: Infrared Remote Sensing Vision-Language Learning with CLIP and VLM Adaptation [12.037405766509274]
  MonoIR-RS is a large-scale infrared remote-sensing vision-language dataset.<n>It couples IR-aware data construction with CLIP-style contrastive adaptation and VLM instruction tuning.<n>We show that the synthesized infrared imagery is markedly closer to real thermal imagery than a grayscale conversion.
  arXiv  Detail & Related papers  (2026-07-07T17:53:58Z)
• InfraNet: Quality-Aware RGB Guidance for Efficient Infrared Object Detection [31.00284151416272]
  InfraNet is a quality-aware framework that regulates RGB guidance during training and supports flexible RGB--IR or IR-only deployment.<n>Our method is evaluated through extensive experiments on four benchmark datasets (LLVIP, FLIR-Aligned, M$3$FD, and DroneVehicle)
  arXiv  Detail & Related papers  (2026-07-04T09:49:53Z)
• Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception [45.307537865864994]
  Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery.<n>We present Thermo-VL, a wavelength-aware VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module.<n>Experiments show strong gains on challenging thermal-only and RGB+thermal reasoning tasks, highlighting the value of prompt-conditioned multispectral fusion.
  arXiv  Detail & Related papers  (2026-05-21T01:43:05Z)
• Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection [45.62297680190076]
  We propose a semantic bridge fusion framework with bi-support modeling for multispectral object detection.<n>Specifically, text is used as a shared semantic bridge to align RGB and IR responses under a unified category condition.<n>We formulate RGB-IR interaction evidence into the regular consensus support and the complementary discrepancy support.
  arXiv  Detail & Related papers  (2026-04-13T09:41:01Z)
• TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation [12.591408054941027]
  TherA is a controllable RGB-to-TIR translation framework that produces diverse and thermally plausible images at both scene and object level.<n>TherA achieves state-of-the-art translation performance, demonstrating improved zero-shot translation performance up to 33% increase averaged across all metrics.
  arXiv  Detail & Related papers  (2026-02-23T01:56:29Z)
• IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark [6.171775609352536]
  We propose IRGPT, the first multi-modal large language model for real-world infrared images.<n>The proposed IR-TD dataset contains real infrared images paired with meticulously handcrafted texts.<n>IRGPT achieves state-of-the-art performance even compared with larger-scale models.
  arXiv  Detail & Related papers  (2025-07-19T02:53:01Z)
• DiffV2IR: Visible-to-Infrared Diffusion Model via Vision-Language Understanding [43.85632218045282]
  We introduce DiffV2IR, a novel framework for image translation comprising two key elements: a Progressive Learning Module (PLM) and a Vision-Language Understanding Module (VLUM)<n>PLM features an adaptive diffusion model architecture that leverages multi-stage knowledge learning to infrared transition from full-range to target wavelength.<n>VLUM incorporates unified Vision-Language Understanding. We also collected a large infrared dataset, IR-500K, which includes 500,000 infrared images compiled by various scenes and objects under various environmental conditions.
  arXiv  Detail & Related papers  (2025-03-24T17:58:09Z)
• Bringing RGB and IR Together: Hierarchical Multi-Modal Enhancement for Robust Transmission Line Detection [67.02804741856512]
  We propose a novel Hierarchical Multi-Modal Enhancement Network (HMMEN) that integrates RGB and IR data for robust and accurate TL detection.<n>Our method introduces two key components: (1) a Mutual Multi-Modal Enhanced Block (MMEB), which fuses and enhances hierarchical RGB and IR feature maps in a coarse-to-fine manner, and (2) a Feature Alignment Block (FAB) that corrects misalignments between decoder outputs and IR feature maps by leveraging deformable convolutions.
  arXiv  Detail & Related papers  (2025-01-25T06:21:06Z)
• SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval [82.51117533271517]
  Previous works typically only encode RGB videos to obtain high-level semantic features. Existing RGB-based sign retrieval works suffer from the huge memory cost of dense visual data embedding in end-to-end training. We propose a novel sign language representation framework called Semantically Enhanced Dual-Stream.
  arXiv  Detail & Related papers  (2024-07-23T11:31:11Z)
• Dual Swin-Transformer based Mutual Interactive Network for RGB-D Salient Object Detection [67.33924278729903]
  In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the long-range dependencies in visual inputs. Comprehensive experiments on five standard RGB-D SOD benchmark datasets demonstrate the superiority of the proposed DTMINet method.
  arXiv  Detail & Related papers  (2022-06-07T08:35:41Z)
• Data-Level Recombination and Lightweight Fusion Scheme for RGB-D Salient Object Detection [73.31632581915201]
  We propose a novel data-level recombination strategy to fuse RGB with D (depth) before deep feature extraction. A newly lightweight designed triple-stream network is applied over these novel formulated data to achieve an optimal channel-wise complementary fusion status between the RGB and D.
  arXiv  Detail & Related papers  (2020-08-07T10:13:05Z)
• Synergistic saliency and depth prediction for RGB-D saliency detection [76.27406945671379]
  Existing RGB-D saliency datasets are small, which may lead to overfitting and limited generalization for diverse scenarios. We propose a semi-supervised system for RGB-D saliency detection that can be trained on smaller RGB-D saliency datasets without saliency ground truth.
  arXiv  Detail & Related papers  (2020-07-03T14:24:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.