RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

• URL: http://arxiv.org/abs/2306.11029v4
• Date: Tue, 16 Apr 2024 11:24:36 GMT
• Title: RemoteCLIP: A Vision Language Foundation Model for Remote Sensing
• Authors: Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, Jun Zhou,
• Abstract summary: We propose RemoteCLIP, a vision-language foundation model for remote sensing. It aims to learn robust visual features with rich semantics and aligned text embeddings for seamless downstream application. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, $textitk$-NN classification, few-shot classification, image-text retrieval, and object counting in remote sensing images.
• Score: 13.814076157988225
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: General-purpose foundation models have led to recent breakthroughs in artificial intelligence. In remote sensing, self-supervised learning (SSL) and Masked Image Modeling (MIM) have been adopted to build foundation models. However, these models primarily learn low-level features and require annotated data for fine-tuning. Moreover, they are inapplicable for retrieval and zero-shot applications due to the lack of language understanding. To address these limitations, we propose RemoteCLIP, the first vision-language foundation model for remote sensing that aims to learn robust visual features with rich semantics and aligned text embeddings for seamless downstream application. To address the scarcity of pre-training data, we leverage data scaling which converts heterogeneous annotations into a unified image-caption data format based on Box-to-Caption (B2C) and Mask-to-Box (M2B) conversion. By further incorporating UAV imagery, we produce a 12 $\times$ larger pretraining dataset than the combination of all available datasets. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, $\textit{k}$-NN classification, few-shot classification, image-text retrieval, and object counting in remote sensing images. Evaluation on 16 datasets, including a newly introduced RemoteCount benchmark to test the object counting ability, shows that RemoteCLIP consistently outperforms baseline foundation models across different model scales. Impressively, RemoteCLIP beats the state-of-the-art method by 9.14% mean recall on the RSITMD dataset and 8.92% on the RSICD dataset. For zero-shot classification, our RemoteCLIP outperforms the CLIP baseline by up to 6.39% average accuracy on 12 downstream datasets. Project website: https://github.com/ChenDelong1999/RemoteCLIP

Related papers

• ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation [5.179738379203527]
  In absence of parallax cues, a learning-based single image depth estimation model relies heavily on shading and contextual cues in the image. It is necessary to train such models on large and varied datasets, which are difficult to capture. Using embeddings from pre-trained foundational models, such as CLIP, improves zero shot transfer in several applications.
  arXiv  Detail & Related papers  (2024-03-27T17:53:30Z)
• Raising the Bar of AI-generated Image Detection with CLIP [50.345365081177555]
  The aim of this work is to explore the potential of pre-trained vision-language models (VLMs) for universal detection of AI-generated images. We develop a lightweight detection strategy based on CLIP features and study its performance in a wide variety of challenging scenarios.
  arXiv  Detail & Related papers  (2023-11-30T21:11:20Z)
• VeCLIP: Improving CLIP Training via Visual-enriched Captions [63.547204530720705]
  This study introduces a scalable pipeline for noisy caption rewriting. We emphasize the incorporation of visual concepts into captions, termed as Visual-enriched Captions (VeCap) We showcase the adaptation of this method for training CLIP on large-scale web-crawled datasets, termed VeCLIP.
  arXiv  Detail & Related papers  (2023-10-11T17:49:13Z)
• Demystifying CLIP Data [86.34045746910114]
  Contrastive Language-Image Pre-training (CLIP) has advanced research and applications in computer vision. We introduce Metadata-Curated Language-Image Pre-training (MetaCLIP) MetaCLIP takes a raw data pool and metadata (derived from CLIP's concepts) and yields a balanced subset over the metadata distribution.
  arXiv  Detail & Related papers  (2023-09-28T17:59:56Z)
• ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation [20.57370550156505]
  ReCLIP is a source-free domain adaptation method for vision-language models. We demonstrate ReCLIP reduces the average error rate of CLIP from 30.17% to 25.06% on 22 image classification benchmarks.
  arXiv  Detail & Related papers  (2023-08-04T18:11:40Z)
• Text2Seg: Remote Sensing Image Semantic Segmentation via Text-Guided Visual Foundation Models [7.452422412106768]
  We propose a novel method named Text2Seg for remote sensing semantic segmentation. It overcomes the dependency on extensive annotations by employing an automatic prompt generation process. We show that Text2Seg significantly improves zero-shot prediction performance compared to the vanilla SAM model.
  arXiv  Detail & Related papers  (2023-04-20T18:39:41Z)
• Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models [13.340759455910721]
  We propose a novel method to prevent zero-shot transfer degradation in the continual learning of vision-language models. Our method outperforms other methods in the traditional class-incremental learning setting.
  arXiv  Detail & Related papers  (2023-03-12T10:28:07Z)
• Prefix Conditioning Unifies Language and Label Supervision [84.11127588805138]
  We show that dataset biases negatively affect pre-training by reducing the generalizability of learned representations. In experiments, we show that this simple technique improves the performance in zero-shot image recognition accuracy and robustness to the image-level distribution shift.
  arXiv  Detail & Related papers  (2022-06-02T16:12:26Z)
• ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension [114.85628613911713]
  Large-scale pre-trained models are useful for image classification across domains. We present ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a state-of-the-art large-scale model, for ReC.
  arXiv  Detail & Related papers  (2022-04-12T17:55:38Z)
• Self-Supervised Pre-Training for Transformer-Based Person Re-Identification [54.55281692768765]
  Transformer-based supervised pre-training achieves great performance in person re-identification (ReID) Due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset to boost the performance. This work aims to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure.
  arXiv  Detail & Related papers  (2021-11-23T18:59:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.