Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data

• URL: http://arxiv.org/abs/2609.03391v1
• Date: Thu, 03 Sep 2026 05:45:31 GMT
• Title: Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data
• Abstract summary: We propose an end-to-end contrastive learning framework that supports multi-source sensor inputs for vision-language modeling.<n>We introduce a spectral-context-aware mask-based contrastive learning scheme to suppress modality-specific redundant features.<n>To support multi-modal training, we construct OmniRS5M, the first large-scale remote sensing image-text corpus.
• Score: 12.302498570831538
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive language-image learning (CLIP) has become a key paradigm for remote sensing vision-language understanding. However, existing remote sensing contrastive learning methods are mostly built on RGB-oriented CLIP architectures, making it difficult to exploit heterogeneous sensors such as SAR, multi-spectral imaging (MSI), and hyperspectral imaging (HSI). To address this limitation, we propose OmniRSCLIP, an end-to-end contrastive learning framework that supports multi-source sensor inputs for remote sensing vision-language modeling. The key idea is to extend CLIP beyond its fixed RGB input interface without breaking the pretrained visual knowledge. To this end, OmniRSCLIP introduces Spectral-Spatial Basis Decomposition (SSBD), which formulates arbitrary-channel adaptation as a basis recomposition problem: pretrained CLIP patch embeddings provide transferable spatial bases, while wavelength-conditioned coefficients span sensor-specific embedding kernels within a constrained visual prior space. This design avoids forcing heterogeneous sensors into a fixed-channel input space, while aligning them in a unified image-text semantic space. We further introduce a spectral-context-aware mask-based contrastive learning scheme to suppress modality-specific redundant features and enhance fine-grained image-text alignment. Finally, to support multi-modal training, we construct OmniRS5M, the first large-scale remote sensing image-text corpus covering RGB, SAR, MSI, and HSI. Experiments on retrieval, zero-shot classification, and semantic localization show that OmniRSCLIP preserves strong RGB-domain performance while effectively extending CLIP to heterogeneous remote sensing modalities.

Related papers

• DPSF-Net: A Dual-Prior Spatial-Frequency Network for Real-World Remote Sensing Image Dehazing [1.0096416727161845]
  We propose DPSF-Net, a dual-prior spatial-frequency network built on MCAF-Net for real-world RSID.<n>The network uses hazy RGB images and dark channel prior (DCP) maps as joint inputs, allowing physical degradation cues to guide end-to-end feature learning.<n>Experiments demonstrate that DPSF-Net achieves state-of-the-art performance on the real-world RRSHID remote sensing image dehazing benchmark.
  arXiv  Detail & Related papers  (2026-09-07T03:00:10Z)
• Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing [55.13802890769086]
  We present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing.<n>To achieve profound modal synergy, Evita embeds a suite of intrinsic co-learning modules directly into every encoder layer.<n>Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.
  arXiv  Detail & Related papers  (2026-07-10T06:52:09Z)
• ExACT: Exemplar-Driven Calibrated Refinement for Training-Free Visual Grounding in Remote Sensing Images [51.53960096934913]
  Remote sensing visual grounding (RSVG) aims to locate specific objects in high-resolution RS imagery using free-form natural language descriptions.<n>Recent advances in multimodal large language models (MLLMs) show great potential for such open-vocabulary RSVG.<n>We propose Exemplar-driven Calibrated Refinement (ExACT), a training-free framework driven by a one-shot visual prompting mechanism.
  arXiv  Detail & Related papers  (2026-06-27T13:50:39Z)
• TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images [48.9420428682499]
  This paper proposes a text semantic-assisted cross-modal image registration framework, named TAR, for optical and SAR images.<n> TAR exploits text semantic priors from remote sensing scenes and land-cover categories to alleviate the modality gap.<n>It achieves stronger matching performance than several state-of-the-art methods and yields significant gains under large geometric deformations.
  arXiv  Detail & Related papers  (2026-05-12T12:50:36Z)
• Deep Spatially-Regularized and Superpixel-Based Diffusion Learning for Unsupervised Hyperspectral Image Clustering [3.702642055407484]
  unsupervised framework for hyperspectral image (HSI) clustering is proposed.<n>Masked deep representation learning with diffusion-based clustering is proposed.<n>Deep Spatially-Regularized Superpixel-based Diffusion Learning ($DS2DL$) algorithm is proposed.
  arXiv  Detail & Related papers  (2026-04-14T21:21:51Z)
• Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining [59.2578488860426]
  Heterogeneous multi-modal remote sensing object detection aims to accurately detect objects from diverse sensors.<n>Existing approaches largely adopt a late alignment paradigm, in which modality alignment and task-specific optimization are entangled during downstream fine-tuning.<n>We propose BabelRS, a unified language-pivoted pretraining framework that explicitly decouples modality alignment from downstream task learning.
  arXiv  Detail & Related papers  (2026-03-02T11:38:12Z)
• Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models [18.24287471339871]
  Hyperspectral imaging (HSI) captures spatial information along with dense spectral measurements across numerous narrow wavelength bands.<n>Our architecture incorporates a spectral transformer and a spectrum-aware spatial prior module to extract rich spatial-spectral features.<n>Our architecture achieves state-of-the-art semantic segmentation performance while directly using HSI inputs, outperforming both vision-based and hyperspectral segmentation methods.
  arXiv  Detail & Related papers  (2025-09-24T13:32:07Z)
• Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images [51.74614065919118]
  This paper introduces SegEarth-OV, the first framework for annotation-free open-vocabulary segmentation of RS images.<n>We propose SimFeatUp, a universal upsampler that robustly restores high-resolution spatial details from coarse features.<n>We also present a simple yet effective Global Bias Alleviation operation to subtract the inherent global context from patch features.
  arXiv  Detail & Related papers  (2025-08-25T14:22:57Z)
• AuxDet: Auxiliary Metadata Matters for Omni-Domain Infrared Small Target Detection [49.81255045696323]
  We present the Auxiliary Metadata Driven Infrared Small Target Detector (AuxDet)<n>AuxDet integrates metadata semantics with visual features, guiding adaptive representation learning for each sample.<n>Experiments on the challenging WideIRSTD-Full benchmark demonstrate that AuxDet consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-21T07:02:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.