Pretrain Where? Investigating How Pretraining Data Diversity Impacts Geospatial Foundation Model Performance

• URL: http://arxiv.org/abs/2604.21104v1
• Date: Wed, 22 Apr 2026 21:43:03 GMT
• Title: Pretrain Where? Investigating How Pretraining Data Diversity Impacts Geospatial Foundation Model Performance
• Abstract summary: Performance differences are largely attributed to the model architecture or input modalities, while the role of the pretraining dataset is rarely studied.<n>We created global and per-continent pretraining datasets and evaluated them on global and per-continent downstream datasets.<n>We found that the pretraining dataset from Europe outperformed global and continent-specific pretraining datasets on both global and local downstream evaluations.
• Score: 15.19997016963026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: New geospatial foundation models introduce a new model architecture and pretraining dataset, often sampled using different notions of data diversity. Performance differences are largely attributed to the model architecture or input modalities, while the role of the pretraining dataset is rarely studied. To address this research gap, we conducted a systematic study on how the geographic composition of pretraining data affects a model's downstream performance. We created global and per-continent pretraining datasets and evaluated them on global and per-continent downstream datasets. We found that the pretraining dataset from Europe outperformed global and continent-specific pretraining datasets on both global and local downstream evaluations. To investigate the factors influencing a pretraining dataset's downstream performance, we analysed 10 pretraining datasets using diversity across continents, biomes, landcover and spectral values. We found that only spectral diversity was strongly correlated with performance, while others were weakly correlated. This finding establishes a new dimension of diversity to be accounted for when creating a high-performing pretraining dataset. We open-sourced 7 new pretraining datasets, pretrained models, and our experimental framework at https://github.com/kerner-lab/pretrain-where.

Related papers

• MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training [15.675086189757769]
  MMEarth-Bench is a collection of five new multimodal environmental tasks with 12 modalities, globally distributed data, and both in- and out-of-distribution test splits.<n>We benchmark a diverse set of pretrained models and find that while (multimodal) pretraining tends to improve model robustness in limited data settings, geographic generalization abilities remain poor.<n>We propose a model-agnostic method for test-time training with multimodal reconstruction (TTT-MMR) that uses all the modalities available at test time as auxiliary tasks.
  arXiv  Detail & Related papers  (2026-02-06T00:48:19Z)
• Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs [25.58407005007563]
  We study the extent to which pretrained Graph Neural Networks can be applied across datasets.<n>We propose an extension to capture feature information while still being feature-agnostic.
  arXiv  Detail & Related papers  (2024-12-23T14:28:56Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training [72.8087629914444]
  We study the impact of the trade-off between the intra-class diversity (the number of samples per class) and the inter-class diversity (the number of classes) of a supervised pre-training dataset. With the size of the pre-training dataset fixed, the best downstream performance comes with a balance on the intra-/inter-class diversity.
  arXiv  Detail & Related papers  (2023-05-20T16:23:50Z)
• Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP [43.7219097444333]
  We introduce a testbed of six publicly available data sources to investigate how pre-training distributions induce robustness in CLIP. We find that the performance of the pre-training data varies substantially across distribution shifts. We find that combining multiple sources does not necessarily yield better models, but rather dilutes the robustness of the best individual data source.
  arXiv  Detail & Related papers  (2022-08-10T18:24:23Z)
• Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data [85.43008636875345]
  We show that diverse representation in training data is key to increasing subgroup performances and achieving population level objectives. Our analysis and experiments describe how dataset compositions influence performance and provide constructive results for using trends in existing data, alongside domain knowledge, to help guide intentional, objective-aware dataset design.
  arXiv  Detail & Related papers  (2021-03-05T00:27:08Z)
• CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems [121.78477833009671]
  We investigate the performance of different summarization models under a cross-dataset setting. A comprehensive study of 11 representative summarization systems on 5 datasets from different domains reveals the effect of model architectures and generation ways.
  arXiv  Detail & Related papers  (2020-10-11T02:19:15Z)
• Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics [118.75207687144817]
  We introduce Data Maps, a model-based tool to characterize and diagnose datasets. We leverage a largely ignored source of information: the behavior of the model on individual instances during training. Our results indicate that a shift in focus from quantity to quality of data could lead to robust models and improved out-of-distribution generalization.
  arXiv  Detail & Related papers  (2020-09-22T20:19:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.