AnySat: An Earth Observation Model for Any Resolutions, Scales, and Modalities

• URL: http://arxiv.org/abs/2412.14123v1
• Date: Wed, 18 Dec 2024 18:11:53 GMT
• Title: AnySat: An Earth Observation Model for Any Resolutions, Scales, and Modalities
• Authors: Guillaume Astruc, Nicolas Gonthier, Clement Mallet, Loic Landrieu,
• Abstract summary: We propose AnySat, a multimodal model based on joint embedding predictive architecture (JEPA) and resolution-adaptive spatial encoders. To demonstrate the advantages of this unified approach, we compile GeoPlex, a collection of $5$ multimodal datasets. We then train a single powerful model on these diverse datasets simultaneously.
• Score: 5.767156832161819
• License:
• Abstract: Geospatial models must adapt to the diversity of Earth observation data in terms of resolutions, scales, and modalities. However, existing approaches expect fixed input configurations, which limits their practical applicability. We propose AnySat, a multimodal model based on joint embedding predictive architecture (JEPA) and resolution-adaptive spatial encoders, allowing us to train a single model on highly heterogeneous data in a self-supervised manner. To demonstrate the advantages of this unified approach, we compile GeoPlex, a collection of $5$ multimodal datasets with varying characteristics and $11$ distinct sensors. We then train a single powerful model on these diverse datasets simultaneously. Once fine-tuned, we achieve better or near state-of-the-art results on the datasets of GeoPlex and $4$ additional ones for $5$ environment monitoring tasks: land cover mapping, tree species identification, crop type classification, change detection, and flood segmentation. The code and models are available at https://github.com/gastruc/AnySat.

Related papers

• EarthView: A Large Scale Remote Sensing Dataset for Self-Supervision [72.84868704100595]
  This paper presents a dataset specifically designed for self-supervision on remote sensing data, intended to enhance deep learning applications on Earth monitoring tasks. The dataset spans 15 tera pixels of global remote-sensing data, combining imagery from a diverse range of sources, including NEON, Sentinel, and a novel release of 1m spatial resolution data from Satellogic. Accompanying the dataset is EarthMAE, a tailored Masked Autoencoder developed to tackle the distinct challenges of remote sensing data.
  arXiv  Detail & Related papers  (2025-01-14T13:42:22Z)
• Multi-Scale and Multimodal Species Distribution Modeling [4.022195138381868]
  Species distribution models (SDMs) aim to predict the distribution of species relating occurrence data with environmental variables. Recent applications of deep learning to SDMs have enabled new avenues, specifically the inclusion of spatial data. We develop a modular structure for SDMs that allows us to test the effect of scale in both single- and multi-scale settings. Results on the GeoLifeCLEF 2023 benchmark indicate that considering multimodal data and learning multi-scale representations leads to more accurate models.
  arXiv  Detail & Related papers  (2024-11-06T15:57:20Z)
• SpectralEarth: Training Hyperspectral Foundation Models at Scale [47.93167977587301]
  We introduce SpectralEarth, a large-scale multi-temporal dataset designed to pretrain hyperspectral foundation models. We pretrain a series of foundation models on SpectralEarth using state-of-the-art self-supervised learning (SSL) algorithms. We construct four downstream datasets for land-cover and crop-type mapping, providing benchmarks for model evaluation.
  arXiv  Detail & Related papers  (2024-08-15T22:55:59Z)
• SatSwinMAE: Efficient Autoencoding for Multiscale Time-series Satellite Imagery [1.6180992915701702]
  We extend the SwinE model to integrate temporal information for satellite time-series data. The architecture employs a hierarchical 3D Masked Autoencoder (MAE) with Video Swin Transformer blocks. Our approach shows significant performance improvements over existing state-of-the-art foundation models for all the evaluated downstream tasks.
  arXiv  Detail & Related papers  (2024-05-03T22:55:56Z)
• UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction [93.77809355002591]
  We introduce UniTraj, a comprehensive framework that unifies various datasets, models, and evaluation criteria. We conduct extensive experiments and find that model performance significantly drops when transferred to other datasets. We provide insights into dataset characteristics to explain these findings.
  arXiv  Detail & Related papers  (2024-03-22T10:36:50Z)
• Fewshot learning on global multimodal embeddings for earth observation tasks [5.057850174013128]
  We pretrain a CLIP/ViT based model using three different modalities of satellite imagery covering over 10% of Earth's total landmass. We use the embeddings produced for each modality with a classical machine learning method to attempt different downstream tasks for earth observation. We visually show that this embedding space, obtained with no labels, is sensible to the different earth features represented by the labelled datasets we selected.
  arXiv  Detail & Related papers  (2023-09-29T20:15:52Z)
• T1: Scaling Diffusion Probabilistic Fields to High-Resolution on Unified Visual Modalities [69.16656086708291]
  Diffusion Probabilistic Field (DPF) models the distribution of continuous functions defined over metric spaces. We propose a new model comprising of a view-wise sampling algorithm to focus on local structure learning. The model can be scaled to generate high-resolution data while unifying multiple modalities.
  arXiv  Detail & Related papers  (2023-05-24T03:32:03Z)
• Heterogenous Ensemble of Models for Molecular Property Prediction [55.91865861896012]
  We propose a method for considering different modalities on molecules. We ensemble these models with a HuberRegressor. This yields a winning solution to the 2textsuperscriptnd edition of the OGB Large-Scale Challenge (2022)
  arXiv  Detail & Related papers  (2022-11-20T17:25:26Z)
• Manifold Topology Divergence: a Framework for Comparing Data Manifolds [109.0784952256104]
  We develop a framework for comparing data manifold, aimed at the evaluation of deep generative models. Based on the Cross-Barcode, we introduce the Manifold Topology Divergence score (MTop-Divergence) We demonstrate that the MTop-Divergence accurately detects various degrees of mode-dropping, intra-mode collapse, mode invention, and image disturbance.
  arXiv  Detail & Related papers  (2021-06-08T00:30:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.