Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data

• URL: http://arxiv.org/abs/2103.16607v1
• Date: Tue, 30 Mar 2021 18:26:39 GMT
• Title: Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data
• Authors: Oscar Ma\~nas, Alexandre Lacoste, Xavier Giro-i-Nieto, David Vazquez, Pau Rodriguez
• Abstract summary: Seasonal Contrast (SeCo) is an effective pipeline to leverage unlabeled data for in-domain pre-training of re-mote sensing representations. SeCo will be made public to facilitate transfer learning and enable rapid progress in re-mote sensing applications.
• Score: 64.40187171234838
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Remote sensing and automatic earth monitoring are key to solve global-scale challenges such as disaster prevention, land use monitoring, or tackling climate change. Although there exist vast amounts of remote sensing data, most of it remains unlabeled and thus inaccessible for supervised learning algorithms. Transfer learning approaches can reduce the data requirements of deep learning algorithms. However, most of these methods are pre-trained on ImageNet and their generalization to remote sensing imagery is not guaranteed due to the domain gap. In this work, we propose Seasonal Contrast (SeCo), an effective pipeline to leverage unlabeled data for in-domain pre-training of re-mote sensing representations. The SeCo pipeline is com-posed of two parts. First, a principled procedure to gather large-scale, unlabeled and uncurated remote sensing datasets containing images from multiple Earth locations at different timestamps. Second, a self-supervised algorithm that takes advantage of time and position invariance to learn transferable representations for re-mote sensing applications. We empirically show that models trained with SeCo achieve better performance than their ImageNet pre-trained counterparts and state-of-the-art self-supervised learning methods on multiple downstream tasks. The datasets and models in SeCo will be made public to facilitate transfer learning and enable rapid progress in re-mote sensing applications.

Related papers

• Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery [78.43828998065071]
  Recent advances in unsupervised learning have demonstrated the ability of large vision models to achieve promising results on downstream tasks. Such pre-training techniques have also been explored recently in the remote sensing domain due to the availability of large amount of unlabelled data. In this paper, we re-visit transformers pre-training and leverage multi-scale information that is effectively utilized with multiple modalities.
  arXiv  Detail & Related papers  (2024-03-08T16:18:04Z)
• Joint multi-modal Self-Supervised pre-training in Remote Sensing: Application to Methane Source Classification [10.952006057356714]
  In earth observation, there are opportunities to exploit domain-specific remote sensing image data. We briefly review the core principles behind so-called joint-embeddings methods and investigate the usage of multiple remote sensing modalities in self-supervised pre-training.
  arXiv  Detail & Related papers  (2023-06-16T14:01:57Z)
• Supervised and Contrastive Self-Supervised In-Domain Representation Learning for Dense Prediction Problems in Remote Sensing [0.0]
  This paper explores the effectiveness of in-domain representations in both supervised and self-supervised forms to solve the domain difference between remote sensing and the ImageNet dataset. For self-supervised pre-training, we have utilized the SimSiam algorithm as it is simple and does not need huge computational resources. Our results have demonstrated that using datasets with a high spatial resolution for self-supervised representation learning leads to high performance in downstream tasks.
  arXiv  Detail & Related papers  (2023-01-29T20:56:51Z)
• Evaluating the Label Efficiency of Contrastive Self-Supervised Learning for Multi-Resolution Satellite Imagery [0.0]
  Self-supervised learning has been applied in the remote sensing domain to exploit readily-available unlabeled data. In this paper, we study self-supervised visual representation learning through the lens of label efficiency.
  arXiv  Detail & Related papers  (2022-10-13T06:54:13Z)
• Consecutive Pretraining: A Knowledge Transfer Learning Strategy with Relevant Unlabeled Data for Remote Sensing Domain [25.84756140221655]
  ConSecutive PreTraining (CSPT) is proposed based on the idea of not stopping pretraining in natural language processing (NLP) The proposed CSPT also can release the huge potential of unlabeled data for task-aware model training. The results show that by utilizing the proposed CSPT for task-aware model training, almost all downstream tasks in RSD can outperform the previous method of supervised pretraining-then-fine-tuning.
  arXiv  Detail & Related papers  (2022-07-08T12:32:09Z)
• Self-supervised Audiovisual Representation Learning for Remote Sensing Data [96.23611272637943]
  We propose a self-supervised approach for pre-training deep neural networks in remote sensing. By exploiting the correspondence between geo-tagged audio recordings and remote sensing, this is done in a completely label-free manner. We show that our approach outperforms existing pre-training strategies for remote sensing imagery.
  arXiv  Detail & Related papers  (2021-08-02T07:50:50Z)
• Geographical Knowledge-driven Representation Learning for Remote Sensing Images [18.79154074365997]
  We propose a Geographical Knowledge-driven Representation learning method for remote sensing images (GeoKR) The global land cover products and geographical location associated with each remote sensing image are regarded as geographical knowledge. A large scale pre-training dataset Levir-KR is proposed to support network pre-training.
  arXiv  Detail & Related papers  (2021-07-12T09:23:15Z)
• Deep Traffic Sign Detection and Recognition Without Target Domain Real Images [52.079665469286496]
  We propose a novel database generation method that requires no real image from the target-domain, and (ii) templates of the traffic signs. The method does not aim at overcoming the training with real data, but to be a compatible alternative when the real data is not available. On large data sets, training with a fully synthetic data set almost matches the performance of training with a real one.
  arXiv  Detail & Related papers  (2020-07-30T21:06:47Z)
• Laplacian Denoising Autoencoder [114.21219514831343]
  We propose to learn data representations with a novel type of denoising autoencoder. The noisy input data is generated by corrupting latent clean data in the gradient domain. Experiments on several visual benchmarks demonstrate that better representations can be learned with the proposed approach.
  arXiv  Detail & Related papers  (2020-03-30T16:52:39Z)
• Understanding Self-Training for Gradual Domain Adaptation [107.37869221297687]
  We consider gradual domain adaptation, where the goal is to adapt an initial classifier trained on a source domain given only unlabeled data that shifts gradually in distribution towards a target domain. We prove the first non-vacuous upper bound on the error of self-training with gradual shifts, under settings where directly adapting to the target domain can result in unbounded error. The theoretical analysis leads to algorithmic insights, highlighting that regularization and label sharpening are essential even when we have infinite data, and suggesting that self-training works particularly well for shifts with small Wasserstein-infinity distance.
  arXiv  Detail & Related papers  (2020-02-26T08:59:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.