Contrastive ground-level image and remote sensing pre-training improves representation learning for natural world imagery

• URL: http://arxiv.org/abs/2409.19439v1
• Date: Sat, 28 Sep 2024 19:07:22 GMT
• Title: Contrastive ground-level image and remote sensing pre-training improves representation learning for natural world imagery
• Authors: Andy V. Huynh, Lauren E. Gillespie, Jael Lopez-Saucedo, Claire Tang, Rohan Sikand, Moisés Expósito-Alonso,
• Abstract summary: In this paper, we show how views of image data with contrast learning can be leveraged. For example, we show how multiple views of image data can be combined to improve classification for species.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal image-text contrastive learning has shown that joint representations can be learned across modalities. Here, we show how leveraging multiple views of image data with contrastive learning can improve downstream fine-grained classification performance for species recognition, even when one view is absent. We propose ContRastive Image-remote Sensing Pre-training (CRISP)$\unicode{x2014}$a new pre-training task for ground-level and aerial image representation learning of the natural world$\unicode{x2014}$and introduce Nature Multi-View (NMV), a dataset of natural world imagery including $>3$ million ground-level and aerial image pairs for over 6,000 plant taxa across the ecologically diverse state of California. The NMV dataset and accompanying material are available at hf.co/datasets/andyvhuynh/NatureMultiView.

Related papers

• AerialMegaDepth: Learning Aerial-Ground Reconstruction and View Synthesis [57.249817395828174]
  We propose a scalable framework combining pseudo-synthetic renderings from 3D city-wide meshes with real, ground-level crowd-sourced images. The pseudo-synthetic data simulates a wide range of aerial viewpoints, while the real, crowd-sourced images help improve visual fidelity for ground-level images. Using this hybrid dataset, we fine-tune several state-of-the-art algorithms and achieve significant improvements on real-world, zero-shot aerial-ground tasks.
  arXiv  Detail & Related papers  (2025-04-17T17:57:05Z)
• Contrasting Deepfakes Diffusion via Contrastive Learning and Global-Local Similarities [88.398085358514]
  Contrastive Deepfake Embeddings (CoDE) is a novel embedding space specifically designed for deepfake detection. CoDE is trained via contrastive learning by additionally enforcing global-local similarities.
  arXiv  Detail & Related papers  (2024-07-29T18:00:10Z)
• Classifying geospatial objects from multiview aerial imagery using semantic meshes [2.116528763953217]
  We propose a new method to predict tree species based on aerial images of forests in the U.S. We show that our proposed multiview method improves classification accuracy from 53% to 75% relative to an orthoorthoaic baseline on a challenging cross-site tree classification task.
  arXiv  Detail & Related papers  (2024-05-15T17:56:49Z)
• ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data [8.905439446173503]
  Vision-language models (VLMs) are generally trained on datasets consisting of image-caption pairs obtained from the web. Real-world multimodal datasets, such as healthcare data, are significantly more complex. ViLLA is trained to capture fine-grained region-attribute relationships from complex datasets.
  arXiv  Detail & Related papers  (2023-08-22T05:03:09Z)
• Toward Real-world Single Image Deraining: A New Benchmark and Beyond [79.5893880599847]
  Single image deraining (SID) in real scenarios attracts increasing attention in recent years. Previous real datasets suffer from low-resolution images, homogeneous rain streaks, limited background variation, and even misalignment of image pairs. We establish a new high-quality dataset named RealRain-1k, consisting of $1,120$ high-resolution paired clean and rainy images with low- and high-density rain streaks, respectively.
  arXiv  Detail & Related papers  (2022-06-11T12:26:59Z)
• Unified Contrastive Learning in Image-Text-Label Space [130.31947133453406]
  Unified Contrastive Learning (UniCL) is effective way of learning semantically rich yet discriminative representations. UniCL stand-alone is a good learner on pure imagelabel data, rivaling the supervised learning methods across three image classification datasets.
  arXiv  Detail & Related papers  (2022-04-07T17:34:51Z)
• Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid [102.24539566851809]
  Restoring reasonable and realistic content for arbitrary missing regions in images is an important yet challenging task. Recent image inpainting models have made significant progress in generating vivid visual details, but they can still lead to texture blurring or structural distortions. We propose the Semantic Pyramid Network (SPN) motivated by the idea that learning multi-scale semantic priors can greatly benefit the recovery of locally missing content in images.
  arXiv  Detail & Related papers  (2021-12-08T04:33:33Z)
• Focus on the Positives: Self-Supervised Learning for Biodiversity Monitoring [9.086207853136054]
  We address the problem of learning self-supervised representations from unlabeled image collections. We exploit readily available context data that encodes information such as the spatial and temporal relationships between the input images. For the critical task of global biodiversity monitoring, this results in image features that can be adapted to challenging visual species classification tasks with limited human supervision.
  arXiv  Detail & Related papers  (2021-08-14T01:12:41Z)
• Curious Representation Learning for Embodied Intelligence [81.21764276106924]
  Self-supervised representation learning has achieved remarkable success in recent years. Yet to build truly intelligent agents, we must construct representation learning algorithms that can learn from environments. We propose a framework, curious representation learning, which jointly learns a reinforcement learning policy and a visual representation model.
  arXiv  Detail & Related papers  (2021-05-03T17:59:20Z)
• Benchmarking Representation Learning for Natural World Image Collections [13.918304838054846]
  We present two new natural world visual classification datasets, iNat2021 and NeWT. The former consists of 2.7M images from 10k different species uploaded by users of the citizen science application iNaturalist. We benchmarking the performance of representation learning algorithms on a suite of challenging natural world binary classification tasks that go beyond standard species classification. We provide a comprehensive analysis of feature extractors trained with and without supervision on ImageNet and iNat2021, shedding light on the strengths and weaknesses of different learned features across a diverse set of tasks.
  arXiv  Detail & Related papers  (2021-03-30T16:41:49Z)
• Free-Form Image Inpainting via Contrastive Attention Network [64.05544199212831]
  In image inpainting tasks, masks with any shapes can appear anywhere in images which form complex patterns. It is difficult for encoders to capture such powerful representations under this complex situation. We propose a self-supervised Siamese inference network to improve the robustness and generalization.
  arXiv  Detail & Related papers  (2020-10-29T14:46:05Z)
• AiRound and CV-BrCT: Novel Multi-View Datasets for Scene Classification [2.931113769364182]
  We present two new publicly available datasets named thedatasetand CV-BrCT. The first one contains triplets of images from the same geographic coordinate with different perspectives of view extracted from various places around the world. The second dataset contains pairs of aerial and street-level images extracted from southeast Brazil.
  arXiv  Detail & Related papers  (2020-08-03T18:55:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.