Scalable Cross-Facility Federated Learning for Scientific Foundation Models on Multiple Supercomputers

• URL: http://arxiv.org/abs/2603.19544v1
• Date: Fri, 20 Mar 2026 00:57:58 GMT
• Title: Scalable Cross-Facility Federated Learning for Scientific Foundation Models on Multiple Supercomputers
• Abstract summary: We present a cross-facility Federated Learning framework for heterogeneous HPC environments.<n>We evaluate it across four U.S. Department of Energy (DOE) leadership-class supercomputers.<n>We validate the scientific applicability by fine-tuning a large language model on a chemistry instruction dataset.
• Score: 10.377831886929803
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Artificial Intelligence for scientific applications increasingly requires training large models on data that cannot be centralized due to privacy constraints, data sovereignty, or the sheer volume of data generated. Federated learning (FL) addresses this by enabling collaborative training without centralizing raw data, but scientific applications demand model scales that requires extensive computing resources, typically offered at High Performance Computing (HPC) facilities. Deploying FL experiments across HPC facilities introduces challenges beyond cloud or enterprise settings. We present a comprehensive cross-facility FL framework for heterogeneous HPC environments, built on Advanced Privacy-Preserving Federated Learning (APPFL) framework with Globus Compute and Transfer orchestration, and evaluate it across four U.S. Department of Energy (DOE) leadership-class supercomputers. We demonstrate that FL experiments across HPC facilities are practically achievable, characterize key sources of heterogeneity impacting the training performance, and show that algorithmic choices matter significantly under realistic HPC scheduling conditions. We validate the scientific applicability by fine-tuning a large language model on a chemistry instruction dataset, and identify scheduler-aware algorithm design as a critical open challenge for future deployments.

Related papers

• QFed: Parameter-Compact Quantum-Classical Federated Learning [7.031234391152914]
  We introduce QFed, a quantum-enabled federated learning framework aimed at boosting computational efficiency across edge device networks.<n> Experimental results show that QFed achieves a 77.6% reduction in the parameter count of a VGG-like model while maintaining an accuracy comparable to classical approaches in a scalable environment.
  arXiv  Detail & Related papers  (2026-01-14T19:16:20Z)
• Federated Learning Framework for Scalable AI in Heterogeneous HPC and Cloud Environments [0.1805840413757548]
  We present a federated learning framework built to run efficiently across mixed HPC and cloud environments.<n>Our system addresses key challenges such as system het- erogeneity, communication overhead, and resource scheduling, while maintaining model accuracy and data privacy.
  arXiv  Detail & Related papers  (2025-11-22T18:39:25Z)
• Physics-Learning AI Datamodel (PLAID) datasets: a collection of physics simulations for machine learning [0.15469999759898032]
  PLAID is a framework for representing and sharing datasets of physics simulations.<n> PLAID defines a unified standard for describing simulation data.<n>We release six datasets under the PLAID standard, covering structural mechanics and computational fluid dynamics.
  arXiv  Detail & Related papers  (2025-05-05T18:59:17Z)
• Federated Learning in Chemical Engineering: A Tutorial on a Framework for Privacy-Preserving Collaboration Across Distributed Data Sources [0.0]
  This work aims to provide the chemical engineering community with an accessible introduction to the discipline.<n>It explores the application of Federated Learning in tasks such as manufacturing optimization, multimodal data integration, and drug discovery.<n>The tutorial was built using key frameworks such as $textttFlower$ and $texttTensorFlow Federated$ and was designed to provide chemical engineers with the right tools to adopt FL.
  arXiv  Detail & Related papers  (2024-11-23T13:16:06Z)
• Enabling High Data Throughput Reinforcement Learning on GPUs: A Domain Agnostic Framework for Data-Driven Scientific Research [90.91438597133211]
  We introduce WarpSci, a framework designed to overcome crucial system bottlenecks in the application of reinforcement learning. We eliminate the need for data transfer between the CPU and GPU, enabling the concurrent execution of thousands of simulations.
  arXiv  Detail & Related papers  (2024-08-01T21:38:09Z)
• Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Ugly [62.473245910234304]
  This paper takes a hardware-centric approach to explore how Large Language Models can be brought to modern edge computing systems. We provide a micro-level hardware benchmark, compare the model FLOP utilization to a state-of-the-art data center GPU, and study the network utilization in realistic conditions.
  arXiv  Detail & Related papers  (2023-10-04T20:27:20Z)
• Semi-Federated Learning: Convergence Analysis and Optimization of A Hybrid Learning Framework [70.83511997272457]
  We propose a semi-federated learning (SemiFL) paradigm to leverage both the base station (BS) and devices for a hybrid implementation of centralized learning (CL) and FL. We propose a two-stage algorithm to solve this intractable problem, in which we provide the closed-form solutions to the beamformers.
  arXiv  Detail & Related papers  (2023-10-04T03:32:39Z)
• Analysis and Optimization of Wireless Federated Learning with Data Heterogeneity [72.85248553787538]
  This paper focuses on performance analysis and optimization for wireless FL, considering data heterogeneity, combined with wireless resource allocation. We formulate the loss function minimization problem, under constraints on long-term energy consumption and latency, and jointly optimize client scheduling, resource allocation, and the number of local training epochs (CRE) Experiments on real-world datasets demonstrate that the proposed algorithm outperforms other benchmarks in terms of the learning accuracy and energy consumption.
  arXiv  Detail & Related papers  (2023-08-04T04:18:01Z)
• FS-Real: Towards Real-World Cross-Device Federated Learning [60.91678132132229]
  Federated Learning (FL) aims to train high-quality models in collaboration with distributed clients while not uploading their local data. There is still a considerable gap between the flourishing FL research and real-world scenarios, mainly caused by the characteristics of heterogeneous devices and its scales. We propose an efficient and scalable prototyping system for real-world cross-device FL, FS-Real.
  arXiv  Detail & Related papers  (2023-03-23T15:37:17Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Federated Ensemble Model-based Reinforcement Learning in Edge Computing [21.840086997141498]
  Federated learning (FL) is a privacy-preserving distributed machine learning paradigm. We propose a novel FRL algorithm that effectively incorporates model-based RL and ensemble knowledge distillation into FL for the first time. Specifically, we utilise FL and knowledge distillation to create an ensemble of dynamics models for clients, and then train the policy by solely using the ensemble model without interacting with the environment.
  arXiv  Detail & Related papers  (2021-09-12T16:19:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.