MV-CC: Mask Enhanced Video Model for Remote Sensing Change Caption

• URL: http://arxiv.org/abs/2410.23946v1
• Date: Thu, 31 Oct 2024 14:02:40 GMT
• Title: MV-CC: Mask Enhanced Video Model for Remote Sensing Change Caption
• Authors: Ruixun Liu, Kaiyu Li, Jiayi Song, Dongwei Sun, Xiangyong Cao,
• Abstract summary: We introduce a novel video model-based paradigm without design of the fusion module. Specifically, we use the off-the-shelf video encoder to simultaneously extract the temporal and spatial features of bi-temporal images. Our proposed method can obtain better performance compared with other state-of-the-art RSICC methods.
• Score: 8.062368743143388
• License:
• Abstract: Remote sensing image change caption (RSICC) aims to provide natural language descriptions for bi-temporal remote sensing images. Since Change Caption (CC) task requires both spatial and temporal features, previous works follow an encoder-fusion-decoder architecture. They use an image encoder to extract spatial features and the fusion module to integrate spatial features and extract temporal features, which leads to increasingly complex manual design of the fusion module. In this paper, we introduce a novel video model-based paradigm without design of the fusion module and propose a Mask-enhanced Video model for Change Caption (MV-CC). Specifically, we use the off-the-shelf video encoder to simultaneously extract the temporal and spatial features of bi-temporal images. Furthermore, the types of changes in the CC are set based on specific task requirements, and to enable the model to better focus on the regions of interest, we employ masks obtained from the Change Detection (CD) method to explicitly guide the CC model. Experimental results demonstrate that our proposed method can obtain better performance compared with other state-of-the-art RSICC methods. The code is available at https://github.com/liuruixun/MV-CC.

Related papers

• Treat Stillness with Movement: Remote Sensing Change Detection via Coarse-grained Temporal Foregrounds Mining [10.830803079863704]
  We revisit the widely adopted bi-temporal images-based framework and propose a novel Coarse-grained Temporal Mining Augmented (CTMA) framework. To be specific, given the bi-temporal images, we first transform them into a video using temporal operations. Then, a set of temporal encoders is adopted to extract the motion features from the obtained video for coarse-grained changed region.
  arXiv  Detail & Related papers  (2024-08-15T11:04:26Z)
• CM-UNet: Hybrid CNN-Mamba UNet for Remote Sensing Image Semantic Segmentation [19.496409240783116]
  We propose CM-UNet, comprising a CNN-based encoder for extracting local image features and a Mamba-based decoder for aggregating and integrating global information. By integrating the CSMamba block and MSAA module, CM-UNet effectively captures the long-range dependencies and multi-scale global contextual information of large-scale remote-sensing images.
  arXiv  Detail & Related papers  (2024-05-17T04:20:12Z)
• Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation [115.09597127418452]
  Latent-Shift is an efficient text-to-video generation method based on a pretrained text-to-image generation model. We show that Latent-Shift achieves comparable or better results while being significantly more efficient.
  arXiv  Detail & Related papers  (2023-04-17T17:57:06Z)
• Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots [78.23772771485635]
  We introduce a method to segment the visual field into independently moving regions, trained with no ground truth or supervision. It consists of an adversarial conditional encoder-decoder architecture based on Slot Attention.
  arXiv  Detail & Related papers  (2023-04-04T00:26:13Z)
• STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition [50.064502884594376]
  We study the problem of human action recognition using motion capture (MoCap) sequences. We propose a novel Spatial-Temporal Mesh Transformer (STMT) to directly model the mesh sequences. The proposed method achieves state-of-the-art performance compared to skeleton-based and point-cloud-based models.
  arXiv  Detail & Related papers  (2023-03-31T16:19:27Z)
• Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring [82.84513669453744]
  Image-text pretrained models, e.g., CLIP, have shown impressive general multi-modal knowledge learned from large-scale image-text data pairs. We revisit temporal modeling in the context of image-to-video knowledge transferring. We present a simple and effective temporal modeling mechanism extending CLIP model to diverse video tasks.
  arXiv  Detail & Related papers  (2023-01-26T14:12:02Z)
• Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation [90.74732705236336]
  Language-queried video actor segmentation aims to predict the pixel-mask of the actor which performs the actions described by a natural language query in the target frames. We propose a collaborative spatial-temporal encoder-decoder framework which contains a 3D temporal encoder over the video clip to recognize the queried actions, and a 2D spatial encoder over the target frame to accurately segment the queried actors.
  arXiv  Detail & Related papers  (2021-05-14T13:27:53Z)
• Motion-Attentive Transition for Zero-Shot Video Object Segmentation [99.44383412488703]
  We present a Motion-Attentive Transition Network (MATNet) for zero-shot object segmentation. An asymmetric attention block, called Motion-Attentive Transition (MAT), is designed within a two-stream encoder. In this way, the encoder becomes deeply internative, allowing for closely hierarchical interactions between object motion and appearance.
  arXiv  Detail & Related papers  (2020-03-09T16:58:42Z)
• CTM: Collaborative Temporal Modeling for Action Recognition [11.467061749436356]
  We propose a Collaborative Temporal Modeling (CTM) block to learn temporal information for action recognition. CTM includes two collaborative paths: a spatial-aware temporal modeling path, and a spatial-unaware temporal modeling path. Experiments on several popular action recognition datasets demonstrate that CTM blocks bring the performance improvements on 2D CNN baselines.
  arXiv  Detail & Related papers  (2020-02-08T12:14:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.