ChangeChat: An Interactive Model for Remote Sensing Change Analysis via Multimodal Instruction Tuning

• URL: http://arxiv.org/abs/2409.08582v1
• Date: Fri, 13 Sep 2024 07:00:44 GMT
• Title: ChangeChat: An Interactive Model for Remote Sensing Change Analysis via Multimodal Instruction Tuning
• Authors: Pei Deng, Wenqian Zhou, Hanlin Wu,
• Abstract summary: We introduce ChangeChat, the first bitemporal vision-language model (VLM) designed specifically for RS change analysis. ChangeChat utilizes multimodal instruction tuning, allowing it to handle complex queries such as change captioning, category-specific quantification, and change localization. Experiments show that ChangeChat offers a comprehensive, interactive solution for RS change analysis, achieving performance comparable to or even better than state-of-the-art (SOTA) methods on specific tasks.
• Score: 0.846600473226587
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Remote sensing (RS) change analysis is vital for monitoring Earth's dynamic processes by detecting alterations in images over time. Traditional change detection excels at identifying pixel-level changes but lacks the ability to contextualize these alterations. While recent advancements in change captioning offer natural language descriptions of changes, they do not support interactive, user-specific queries. To address these limitations, we introduce ChangeChat, the first bitemporal vision-language model (VLM) designed specifically for RS change analysis. ChangeChat utilizes multimodal instruction tuning, allowing it to handle complex queries such as change captioning, category-specific quantification, and change localization. To enhance the model's performance, we developed the ChangeChat-87k dataset, which was generated using a combination of rule-based methods and GPT-assisted techniques. Experiments show that ChangeChat offers a comprehensive, interactive solution for RS change analysis, achieving performance comparable to or even better than state-of-the-art (SOTA) methods on specific tasks, and significantly surpassing the latest general-domain model, GPT-4. Code and pre-trained weights are available at https://github.com/hanlinwu/ChangeChat.

Related papers

• Show Me What and Where has Changed? Question Answering and Grounding for Remote Sensing Change Detection [82.65760006883248]
  We introduce a new task named Change Detection Question Answering and Grounding (CDQAG) CDQAG extends the traditional change detection task by providing interpretable textual answers and intuitive visual evidence. Based on this, we present VisTA, a simple yet effective baseline method that unifies the tasks of question answering and grounding.
  arXiv  Detail & Related papers  (2024-10-31T11:20:13Z)
• Enhancing Perception of Key Changes in Remote Sensing Image Change Captioning [49.24306593078429]
  We propose a novel framework for remote sensing image change captioning, guided by Key Change Features and Instruction-tuned (KCFI) KCFI includes a ViTs encoder for extracting bi-temporal remote sensing image features, a key feature perceiver for identifying critical change areas, and a pixel-level change detection decoder. To validate the effectiveness of our approach, we compare it against several state-of-the-art change captioning methods on the LEVIR-CC dataset.
  arXiv  Detail & Related papers  (2024-09-19T09:33:33Z)
• Multi-Granularity Language-Guided Multi-Object Tracking [95.91263758294154]
  We propose a new multi-object tracking framework, named LG-MOT, that explicitly leverages language information at different levels of granularity. At inference, our LG-MOT uses the standard visual features without relying on annotated language descriptions. Our LG-MOT achieves an absolute gain of 2.2% in terms of target object association (IDF1 score) compared to the baseline using only visual features.
  arXiv  Detail & Related papers  (2024-06-07T11:18:40Z)
• ChangeBind: A Hybrid Change Encoder for Remote Sensing Change Detection [16.62779899494721]
  Change detection (CD) is a fundamental task in remote sensing (RS) which aims to detect the semantic changes between the same geographical regions at different time stamps. We propose an effective Siamese-based framework to encode the semantic changes occurring in the bi-temporal RS images.
  arXiv  Detail & Related papers  (2024-04-26T17:47:14Z)
• Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysis [28.3763053922823]
  Current RSICI technology encompasses change detection and change captioning, each with its limitations in providing comprehensive interpretation. We propose an interactive Change-Agent, which can follow user instructions to achieve comprehensive change interpretation. The Change-Agent integrates a multi-level change interpretation (MCI) model as the eyes and a large language model (LLM) as the brain.
  arXiv  Detail & Related papers  (2024-03-28T17:55:42Z)
• Advanced Feature Manipulation for Enhanced Change Detection Leveraging Natural Language Models [2.2933109484655794]
  Large language models (LLMs) have been utilized in various domains for their exceptional feature extraction capabilities. In this study, we harness the power of a pre-trained LLM, extracting feature maps from extensive datasets, and employ an auxiliary network to detect changes.
  arXiv  Detail & Related papers  (2024-03-23T22:07:32Z)
• OSCaR: Object State Captioning and State Change Representation [52.13461424520107]
  This paper introduces the Object State Captioning and State Change Representation (OSCaR) dataset and benchmark. OSCaR consists of 14,084 annotated video segments with nearly 1,000 unique objects from various egocentric video collections. It sets a new testbed for evaluating multimodal large language models (MLLMs)
  arXiv  Detail & Related papers  (2024-02-27T01:48:19Z)
• MS-Former: Memory-Supported Transformer for Weakly Supervised Change Detection with Patch-Level Annotations [50.79913333804232]
  We propose a memory-supported transformer (MS-Former) for weakly supervised change detection. MS-Former consists of a bi-directional attention block (BAB) and a patch-level supervision scheme (PSS) Experimental results on three benchmark datasets demonstrate the effectiveness of our proposed method in the change detection task.
  arXiv  Detail & Related papers  (2023-11-16T09:57:29Z)
• Neighborhood Contrastive Transformer for Change Captioning [80.10836469177185]
  We propose a neighborhood contrastive transformer to improve the model's perceiving ability for various changes under different scenes. The proposed method achieves the state-of-the-art performance on three public datasets with different change scenarios.
  arXiv  Detail & Related papers  (2023-03-06T14:39:54Z)
• Changer: Feature Interaction is What You Need for Change Detection [6.385385687682811]
  Change detection is an important tool for long-term earth observation missions. We propose a novel general change detection architecture, MetaChanger, which includes a series of alternative interaction layers in the feature extractor. We observe Changer series models achieve competitive performance on different scale change detection datasets.
  arXiv  Detail & Related papers  (2022-09-17T09:13:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.