How do Self-Supervised Remote Sensing Vision Models Transfer to Downstream Tasks?

• URL: http://arxiv.org/abs/2606.13896v1
• Date: Thu, 11 Jun 2026 20:39:34 GMT
• Title: How do Self-Supervised Remote Sensing Vision Models Transfer to Downstream Tasks?
• Abstract summary: Self-supervised geospatial foundation models learn transferable representations from remote sensing data.<n>We study six representative GeoFMs spanning joint-embedding, reconstruction, and multimodal pretraining families.<n>We find that model rankings change across tasks and adaptation settings.
• Score: 1.5713647046527652
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised geospatial foundation models (GeoFMs) learn transferable representations from remote sensing data, but their downstream behavior is difficult to characterize. We study six representative GeoFMs spanning joint-embedding, reconstruction, and multimodal pretraining families, and evaluate transfer across classification, regression, and segmentation benchmarks under different label availability and downstream pipelines. We find that model rankings change across tasks and adaptation settings. Layerwise probing shows that, in most cases, task-relevant information is more accessible in intermediate transformer blocks compared to final-layer embeddings, and that GeoFMs exhibit distinct depthwise profiles. In segmentation case studies on PASTIS and Sen1Floods11, downstream adaptation settings such as decoder design and fine-tuning can be as impactful as the choice of GeoFM, and standard dense-prediction heads may be poorly aligned with how GeoFMs organize information over depth. Finally, CKA analysis on case studies shows that fine-tuning does not rewrite GeoFMs uniformly across depth, and the strongest changes are localized to the first linear layer of the MLP in ViT blocks. These results help explain why GeoFM rankings shift across benchmarks and motivate more representation-aware evaluation and adaptation strategies.

Related papers

• Scalable and Trustworthy Earth Observation Foundation Models [0.0]
  Foundation models (FMs) have transformed machine learning from isolated task-specific models to general-purpose models pretrained on broad data.<n>This chapter defines the FMs paradigm in remote sensing (RS), then synthesizes the current model landscape.<n>The chapter also incorporates recent benchmark evidence showing that no single geospatial foundation model is universally best.
  arXiv  Detail & Related papers  (2026-07-08T14:31:02Z)
• Earth System Foundation Model (ESFM): A unified framework for heterogeneous data integration and forecasting [52.53742727060265]
  Foundation models (FMs) for the Earth system learn statistical relationships between physical variables across massive datasets.<n>Here, we introduce Earth System Foundation Model (ESFM), a fully open model building on the 3Dwin UNet backbone of the pioneering model.<n>We present findings using dense gridded data (ERA5, CMIP6), regionally masked dense data, sparse gridded MODIS satellite data, and station data.<n>Results demonstrate competitive or superior performance relative to state-of-the-art benchmarks.
  arXiv  Detail & Related papers  (2026-04-20T11:40:39Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• OSM-based Domain Adaptation for Remote Sensing VLMs [49.91326341200221]
  Vision-Language Models (VLMs) adapted to remote sensing rely heavily on domain-specific image-text supervision.<n>We propose OSMDA: a self-contained domain adaptation framework that eliminates this dependency.<n>We conduct exhaustive evaluations spanning 10 benchmarks across image-text-to-text tasks.
  arXiv  Detail & Related papers  (2026-03-12T11:08:30Z)
• How To Embed Matters: Evaluation of EO Embedding Design Choices [3.21036640989662]
  We show how backbone architecture, pretraining strategy, representation depth, spatial aggregation, and representation combination influence EO task performance.<n>We find consistent trends: transformer backbones with mean pooling provide strong default embeddings, intermediate ResNet layers can outperform final layers, self-supervised objectives exhibit task-specific strengths, and combining embeddings from different objectives often improves.
  arXiv  Detail & Related papers  (2026-03-11T11:16:42Z)
• GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI [52.13138825802668]
  GeoFMs are transforming Earth Observation, but evaluation lacks standardized protocols.<n> GEO-Bench-2 addresses this with a comprehensive framework spanning classification, segmentation, regression, object detection, and instance segmentation.<n>Code, data, and leaderboard for GEO-Bench-2 are publicly released under a permissive license.
  arXiv  Detail & Related papers  (2025-11-19T17:45:02Z)
• Landslide Hazard Mapping with Geospatial Foundation Models: Geographical Generalizability, Data Scarcity, and Band Adaptability [11.01843362076594]
  We present a framework for adapting geospatial foundation models (GeoFMs) for landslide mapping.<n>We show that Prithvi-EO-2.0 consistently outperforms task-specific CNNs (U-Net, U-Net++), vision transformers (Segformer, SwinV2-B), and other GeoFMs.<n>We highlight remaining challenges such as computational cost and the limited availability of reusable AI-ready training data for landslide research.
  arXiv  Detail & Related papers  (2025-11-06T15:47:37Z)
• Parameter-Efficient Adaptation of Geospatial Foundation Models through Embedding Deflection [3.805591882842849]
  DEFLECT is a novel strategy for adapting GFMs to multispectral satellite imagery with very few additional parameters.<n>It achieves on-par or higher accuracy with 5-10$times$ fewer parameters for classification and segmentation tasks.
  arXiv  Detail & Related papers  (2025-03-12T15:53:58Z)
• GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Tasks [84.86699025256705]
  We present GEOBench-VLM, a benchmark specifically designed to evaluate Vision-Language Models (VLMs) on geospatial tasks.<n>Our benchmark features over 10,000 manually verified instructions and spanning diverse visual conditions, object types, and scales.<n>We evaluate several state-of-the-art VLMs to assess performance on geospatial-specific challenges.
  arXiv  Detail & Related papers  (2024-11-28T18:59:56Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.