How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?

• URL: http://arxiv.org/abs/2501.12535v1
• Date: Tue, 21 Jan 2025 22:57:09 GMT
• Title: How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?
• Authors: Mirali Purohit, Gedeon Muhawenayo, Esther Rolf, Hannah Kerner,
• Abstract summary: Geospatial Foundation Models (GFMs) can help address global challenges such as climate change, agriculture, and disaster response. Previous work on GFMs focused on tailoring model architecture and pre-text tasks, and did not investigate the impact of pre-training data selection on model performance. Our research explores how the geographic distribution of pre-training data affects the performance of GFMs.
• Score: 10.126199683760344
• License:
• Abstract: Foundation models have made rapid advances in many domains including Earth observation, where Geospatial Foundation Models (GFMs) can help address global challenges such as climate change, agriculture, and disaster response. Previous work on GFMs focused on tailoring model architecture and pre-text tasks, and did not investigate the impact of pre-training data selection on model performance. However, recent works from other domains show that the pre-training data distribution is an important factor influencing the performance of the foundation models. With this motivation, our research explores how the geographic distribution of pre-training data affects the performance of GFMs. We evaluated several pre-training data distributions by sampling different compositions from a global data pool. Our experiments with two GFMs on downstream tasks indicate that balanced and globally representative data compositions often outperform region-specific sampling, highlighting the importance of diversity and global coverage in pre-training data. Our results suggest that the most appropriate data sampling technique may depend on the specific GFM architecture. These findings will support the development of robust GFMs by incorporating quality pre-training data distributions, ultimately improving machine learning solutions for Earth observation.

Related papers

• PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models [3.9031647202359667]
  PANGAEA is a standardized evaluation protocol that covers a diverse set of datasets, tasks, resolutions, sensor modalities, and temporalities. We evaluate the most popular GFMs openly available on this benchmark and analyze their performance across several domains. Our findings highlight the limitations of GFMs, under different scenarios, showing that they do not consistently outperform supervised models.
  arXiv  Detail & Related papers  (2024-12-05T14:40:41Z)
• Tackling Data Heterogeneity in Federated Time Series Forecasting [61.021413959988216]
  Time series forecasting plays a critical role in various real-world applications, including energy consumption prediction, disease transmission monitoring, and weather forecasting. Most existing methods rely on a centralized training paradigm, where large amounts of data are collected from distributed devices to a central cloud server. We propose a novel framework, Fed-TREND, to address data heterogeneity by generating informative synthetic data as auxiliary knowledge carriers.
  arXiv  Detail & Related papers  (2024-11-24T04:56:45Z)
• Local vs. Global Models for Hierarchical Forecasting [0.0]
  This study explores the influence of distinct information utilisation on the accuracy of hierarchical forecasts. We develop Global Forecasting Models (GFMs) to exploit cross-series and cross-hierarchies information. Two specific GFMs based on LightGBM are introduced, demonstrating superior accuracy and lower model complexity.
  arXiv  Detail & Related papers  (2024-11-10T08:51:49Z)
• Enhancing Data Quality in Federated Fine-Tuning of Foundation Models [54.757324343062734]
  We propose a data quality control pipeline for federated fine-tuning of foundation models. This pipeline computes scores reflecting the quality of training data and determines a global threshold for a unified standard. Our experiments show that the proposed quality control pipeline facilitates the effectiveness and reliability of the model training, leading to better performance.
  arXiv  Detail & Related papers  (2024-03-07T14:28:04Z)
• Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning [16.684749528240587]
  Federated Learning (FL) is a distributed training paradigm that enables clients scattered across the world to cooperatively learn a global model without divulging confidential data. FL faces a significant challenge in the form of heterogeneous data distributions among clients, which leads to a reduction in performance and robustness. We introduce foundation model distillation to assist in the federated training of lightweight client models and increase their performance under heterogeneous data settings while keeping inference costs low.
  arXiv  Detail & Related papers  (2023-11-14T19:10:56Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Evaluating and Incentivizing Diverse Data Contributions in Collaborative Learning [89.21177894013225]
  For a federated learning model to perform well, it is crucial to have a diverse and representative dataset. We show that the statistical criterion used to quantify the diversity of the data, as well as the choice of the federated learning algorithm used, has a significant effect on the resulting equilibrium. We leverage this to design simple optimal federated learning mechanisms that encourage data collectors to contribute data representative of the global population.
  arXiv  Detail & Related papers  (2023-06-08T23:38:25Z)
• An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation [91.62129090006745]
  This paper studies the distribution shift problem from the perspective of pre-training and data augmentation. We provide the first comprehensive empirical study focusing on pre-training and data augmentation.
  arXiv  Detail & Related papers  (2022-05-25T13:04:53Z)
• DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixup [58.894901088797376]
  federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data. We propose a general framework to solve the above two challenges simultaneously. We provide comprehensive theoretical analysis including robustness analysis, convergence analysis, and generalization ability.
  arXiv  Detail & Related papers  (2022-04-16T08:08:29Z)
• Meta-Learning for Few-Shot Land Cover Classification [3.8529010979482123]
  We evaluate the model-agnostic meta-learning (MAML) algorithm on classification and segmentation tasks. We find that few-shot model adaptation outperforms pre-training with regular gradient descent. This indicates that model optimization with meta-learning may benefit tasks in the Earth sciences.
  arXiv  Detail & Related papers  (2020-04-28T09:42:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.