TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images

• URL: http://arxiv.org/abs/2605.12064v1
• Date: Tue, 12 May 2026 12:50:36 GMT
• Title: TAR: Text Semantic Assisted Cross-modal Image Registration Framework for Optical and SAR Images
• Abstract summary: This paper proposes a text semantic-assisted cross-modal image registration framework, named TAR, for optical and SAR images.<n> TAR exploits text semantic priors from remote sensing scenes and land-cover categories to alleviate the modality gap.<n>It achieves stronger matching performance than several state-of-the-art methods and yields significant gains under large geometric deformations.
• Score: 48.9420428682499
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing deep learning-based methods can capture shared features from optical and synthetic aperture radar (SAR) images for spatial alignment. However, optical-SAR registration remains challenging under large geometric deformations, because the model needs to simultaneously handle cross-modal appearance discrepancies and complex spatial transformations. To address this issue, this paper proposes a text semantic-assisted cross-modal image registration framework, named TAR, for optical and SAR images. TAR exploits text semantic priors from remote sensing scenes and land-cover categories to alleviate the modality gap and enhance cross-modal feature learning. TAR consists of three components: a multi-scale visual feature learning (MSFL) module, a text-assisted feature enhancement (TAFE) module, and a coarse-to-fine dense matching (CFDM) module. MSFL extracts multi-scale visual features from optical and SAR images. TAFE constructs text descriptors related to remote sensing scenes and land-cover objects, and uses a frozen RemoteCLIP text encoder to extract text features. These text features are introduced through visual-text interaction to enhance high-level visual features for more reliable coarse matching. CFDM then establishes coarse correspondences based on the enhanced high-level features and refines the matched locations using low-level features. Experimental results on cross-modal remote sensing images demonstrate the effectiveness of TAR, which achieves stronger matching performance than several state-of-the-art methods and yields significant gains under large geometric deformations.

Related papers

• Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers [56.76198904599581]
  Text-to-image diffusion models excel at translating language prompts into implicitly grounding concepts through their cross-modal attention mechanisms.<n>Recent multi-modal diffusion transformers extend this by introducing joint self-attentiond image and text tokens, enabling richer and more scalable cross-modal alignment.<n>We introduce Seg4Diff, a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image.
  arXiv  Detail & Related papers  (2025-09-22T17:59:54Z)
• Visual Semantic Description Generation with MLLMs for Image-Text Matching [7.246705430021142]
  We propose a novel framework that bridges the modality gap by leveraging multimodal large language models (MLLMs) as visual semantics.<n>Our approach combines: (1) Instance-level alignment by fusing visual features with VSD to enhance the linguistic expressiveness of image representations, and (2) Prototype-level alignment through VSD clustering to ensure category-level consistency.
  arXiv  Detail & Related papers  (2025-07-11T13:38:01Z)
• ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification [52.405499816861635]
  Multiple instance learning (MIL)-based framework has become the mainstream for processing the whole slide image (WSI)<n>We propose a dual-scale vision-language multiple instance learning (ViLa-MIL) framework for whole slide image classification.
  arXiv  Detail & Related papers  (2025-02-12T13:28:46Z)
• Natural Language Supervision for Low-light Image Enhancement [0.0]
  We introduce a Natural Language Supervision (NLS) strategy, which learns feature maps from text corresponding to images.<n>We also design a Textual Guidance Conditioning Mechanism (TCM) that incorporates the connections between image regions and sentence words.<n>In order to effectively identify and merge features from various levels of image and textual information, we design an Information Fusion Attention (IFA) module.
  arXiv  Detail & Related papers  (2025-01-11T13:53:10Z)
• Scale-wise Bidirectional Alignment Network for Referring Remote Sensing Image Segmentation [12.893224628061516]
  The goal of referring remote sensing image segmentation (RRSIS) is to extract specific pixel-level regions within an aerial image via a natural language expression.<n>We propose an innovative framework called Scale-wise Bidirectional Alignment Network (SBANet) to address these challenges.<n>Our proposed method achieves superior performance in comparison to previous state-of-the-art methods on the RRSIS-D and RefSegRS datasets.
  arXiv  Detail & Related papers  (2025-01-01T14:24:04Z)
• Exploring Fine-Grained Image-Text Alignment for Referring Remote Sensing Image Segmentation [27.13782704236074]
  We propose a new referring remote sensing image segmentation method to fully exploit the visual and linguistic representations.<n>The proposed fine-grained image-text alignment module (FIAM) would simultaneously leverage the features of the input image and the corresponding texts.<n>We evaluate the effectiveness of the proposed method on two public referring remote sensing datasets including RefSegRS and RRSIS-D.
  arXiv  Detail & Related papers  (2024-09-20T16:45:32Z)
• FSMR: A Feature Swapping Multi-modal Reasoning Approach with Joint Textual and Visual Clues [20.587249765287183]
  Feature Swapping Multi-modal Reasoning (FSMR) model is designed to enhance multi-modal reasoning through feature swapping. FSMR incorporates a multi-modal cross-attention mechanism, facilitating the joint modeling of textual and visual information. Experiments on the PMR dataset demonstrate FSMR's superiority over state-of-the-art baseline models.
  arXiv  Detail & Related papers  (2024-03-29T07:28:50Z)
• Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioning [49.48946808024608]
  We propose a novel two-stage vision-language pre-training-based approach to bootstrap interactive image-text alignment for remote sensing image captioning, called BITA. Specifically, the first stage involves preliminary alignment through image-text contrastive learning. In the second stage, the interactive Fourier Transformer connects the frozen image encoder with a large language model.
  arXiv  Detail & Related papers  (2023-12-02T17:32:17Z)
• CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
  We propose an end-to-end CLIP-Driven Referring Image framework (CRIS) CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-11-30T07:29:08Z)
• TediGAN: Text-Guided Diverse Face Image Generation and Manipulation [52.83401421019309]
  TediGAN is a framework for multi-modal image generation and manipulation with textual descriptions. StyleGAN inversion module maps real images to the latent space of a well-trained StyleGAN. visual-linguistic similarity learns the text-image matching by mapping the image and text into a common embedding space. instance-level optimization is for identity preservation in manipulation.
  arXiv  Detail & Related papers  (2020-12-06T16:20:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.