SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning

• URL: http://arxiv.org/abs/2511.21420v1
• Date: Wed, 26 Nov 2025 14:11:19 GMT
• Title: SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning
• Authors: Futian Wang, Mengqi Wang, Xiao Wang, Haowen Wang, Jin Tang,
• Abstract summary: This paper explores the use of the SAM (Segment Anything Model) foundation model to extract region-level representations and inject region-of-interest knowledge into the captioning framework.<n>We employ a CNN/Transformer model to extract global-level vision features, leverage the SAM foundation model to delineate semantic- and motion-level change regions, and utilize a specially constructed knowledge graph to provide information about objects of interest.
• Score: 16.276257222647136
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Remote sensing change captioning is an emerging and popular research task that aims to describe, in natural language, the content of interest that has changed between two remote sensing images captured at different times. Existing methods typically employ CNNs/Transformers to extract visual representations from the given images or incorporate auxiliary tasks to enhance the final results, with weak region awareness and limited temporal alignment. To address these issues, this paper explores the use of the SAM (Segment Anything Model) foundation model to extract region-level representations and inject region-of-interest knowledge into the captioning framework. Specifically, we employ a CNN/Transformer model to extract global-level vision features, leverage the SAM foundation model to delineate semantic- and motion-level change regions, and utilize a specially constructed knowledge graph to provide information about objects of interest. These heterogeneous sources of information are then fused via cross-attention, and a Transformer decoder is used to generate the final natural language description of the observed changes. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple widely used benchmark datasets. The source code of this paper will be released on https://github.com/Event-AHU/SAM_ChangeCaptioning

Related papers

• Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey [86.96983249116614]
  Image-Language Foundation Models (ILFM) have demonstrated remarkable success in image-text understanding/generation tasks.<n>This survey provides the first comprehensive review of this emerging field.
  arXiv  Detail & Related papers  (2025-10-12T15:56:02Z)
• Scale-wise Bidirectional Alignment Network for Referring Remote Sensing Image Segmentation [12.893224628061516]
  The goal of referring remote sensing image segmentation (RRSIS) is to extract specific pixel-level regions within an aerial image via a natural language expression.<n>We propose an innovative framework called Scale-wise Bidirectional Alignment Network (SBANet) to address these challenges.<n>Our proposed method achieves superior performance in comparison to previous state-of-the-art methods on the RRSIS-D and RefSegRS datasets.
  arXiv  Detail & Related papers  (2025-01-01T14:24:04Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Salient Object Detection in Optical Remote Sensing Images Driven by Transformer [69.22039680783124]
  We propose a novel Global Extraction Local Exploration Network (GeleNet) for Optical Remote Sensing Images (ORSI-SOD) Specifically, GeleNet first adopts a transformer backbone to generate four-level feature embeddings with global long-range dependencies. Extensive experiments on three public datasets demonstrate that the proposed GeleNet outperforms relevant state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-09-15T07:14:43Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)
• RRSIS: Referring Remote Sensing Image Segmentation [25.538406069768662]
  Localizing desired objects from remote sensing images is of great use in practical applications. Referring image segmentation, which aims at segmenting out the objects to which a given expression refers, has been extensively studied in natural images. We introduce referring remote sensing image segmentation (RRSIS) to fill in this gap and make some insightful explorations.
  arXiv  Detail & Related papers  (2023-06-14T16:40:19Z)
• Learning Visual Affordance Grounding from Demonstration Videos [76.46484684007706]
  Affordance grounding aims to segment all possible interaction regions between people and objects from an image/video. We propose a Hand-aided Affordance Grounding Network (HAGNet) that leverages the aided clues provided by the position and action of the hand in demonstration videos.
  arXiv  Detail & Related papers  (2021-08-12T11:45:38Z)
• Transformed ROIs for Capturing Visual Transformations in Videos [31.88528313257094]
  We present TROI, a plug-and-play module for CNNs to reason between mid-level feature representations that are otherwise separated in space and time. We achieve state-of-the-art action recognition results on the large-scale datasets Something-Something-V2 and Epic-Kitchens-100.
  arXiv  Detail & Related papers  (2021-06-06T15:59:53Z)
• Exploring Explicit and Implicit Visual Relationships for Image Captioning [11.82805641934772]
  In this paper, we explore explicit and implicit visual relationships to enrich region-level representations for image captioning. Explicitly, we build semantic graph over object pairs and exploit gated graph convolutional networks (Gated GCN) to selectively aggregate local neighbors' information. Implicitly, we draw global interactions among the detected objects through region-based bidirectional encoder representations from transformers.
  arXiv  Detail & Related papers  (2021-05-06T01:47:51Z)
• Video-based Facial Expression Recognition using Graph Convolutional Networks [57.980827038988735]
  We introduce a Graph Convolutional Network (GCN) layer into a common CNN-RNN based model for video-based facial expression recognition. We evaluate our method on three widely-used datasets, CK+, Oulu-CASIA and MMI, and also one challenging wild dataset AFEW8.0.
  arXiv  Detail & Related papers  (2020-10-26T07:31:51Z)
• Exploring and Distilling Cross-Modal Information for Image Captioning [47.62261144821135]
  We argue that such understanding requires visual attention to correlated image regions and semantic attention to coherent attributes of interest. Based on the Transformer, we propose the Global-and-Local Information Exploring-and-Distilling approach that explores and distills the source information in vision and language. Our Transformer-based model achieves a CIDEr score of 129.3 in offline COCO evaluation on the COCO testing set with remarkable efficiency in terms of accuracy, speed, and parameter budget.
  arXiv  Detail & Related papers  (2020-02-28T07:46:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.