Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights

• URL: http://arxiv.org/abs/2509.05142v1
• Date: Fri, 05 Sep 2025 14:34:19 GMT
• Title: Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights
• Authors: Cosmin-Andrei Hatfaludi, Alex Serban,
• Abstract summary: Federated learning has the potential to unlock siloed data and distributed resources by enabling collaborative model training without sharing private data.<n>As more complex foundational models gain widespread use, the need to expand training resources and integrate privately owned data grows as well.<n>We explore the intersection of federated learning and foundational models, aiming to identify, categorize, and characterize technical methods that integrate the two paradigms.
• Score: 0.42970700836450487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning has the potential to unlock siloed data and distributed resources by enabling collaborative model training without sharing private data. As more complex foundational models gain widespread use, the need to expand training resources and integrate privately owned data grows as well. In this article, we explore the intersection of federated learning and foundational models, aiming to identify, categorize, and characterize technical methods that integrate the two paradigms. As a unified survey is currently unavailable, we present a literature survey structured around a novel taxonomy that follows the development life-cycle stages, along with a technical comparison of available methods. Additionally, we provide practical insights and guidelines for implementing and evolving these methods, with a specific focus on the healthcare domain as a case study, where the potential impact of federated learning and foundational models is considered significant. Our survey covers multiple intersecting topics, including but not limited to federated learning, self-supervised learning, fine-tuning, distillation, and transfer learning. Initially, we retrieved and reviewed a set of over 4,200 articles. This collection was narrowed to more than 250 thoroughly reviewed articles through inclusion criteria, featuring 42 unique methods. The methods were used to construct the taxonomy and enabled their comparison based on complexity, efficiency, and scalability. We present these results as a self-contained overview that not only summarizes the state of the field but also provides insights into the practical aspects of adopting, evolving, and integrating foundational models with federated learning.

Related papers

• A Survey on Generative Recommendation: Data, Model, and Tasks [55.36322811257545]
  generative recommendation reconceptualizes recommendation as a generation task rather than discriminative scoring.<n>This survey provides a comprehensive examination through a unified tripartite framework spanning data, model, and task dimensions.<n>We identify five key advantages: world knowledge integration, natural language understanding, reasoning capabilities, scaling laws, and creative generation.
  arXiv  Detail & Related papers  (2025-10-31T04:02:58Z)
• UNIFORM: Unifying Knowledge from Large-scale and Diverse Pre-trained Models [62.76435672183968]
  We introduce a novel framework, namely UNIFORM, for knowledge transfer from a diverse set of off-the-shelf models into one student model.<n>We propose a dedicated voting mechanism to capture the consensus of knowledge both at the logit level and at the feature level.<n>Experiments demonstrate that UNIFORM effectively enhances unsupervised object recognition performance compared to strong knowledge transfer baselines.
  arXiv  Detail & Related papers  (2025-08-27T00:56:11Z)
• A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction [21.966560704390716]
  We review current research on Generative Model Unlearning (GenMU)<n>We propose a unified analytical framework for categorizing unlearning objectives, methodological strategies, and evaluation metrics.<n>We highlight the potential practical value of unlearning techniques in real-world applications.
  arXiv  Detail & Related papers  (2025-07-26T09:49:57Z)
• Multi-objective methods in Federated Learning: A survey and taxonomy [2.519319150166215]
  We propose a first taxonomy on the use of multi-objective methods in connection with Federated Learning.<n>We outline open challenges and possible directions for further research.
  arXiv  Detail & Related papers  (2025-02-05T12:06:43Z)
• Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities [89.40778301238642]
  Model merging is an efficient empowerment technique in the machine learning community. There is a significant gap in the literature regarding a systematic and thorough review of these techniques.
  arXiv  Detail & Related papers  (2024-08-14T16:58:48Z)
• Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations [25.261572089655264]
  Key challenge is to efficiently train models across multiple clients with different data distributions, model structures, task objectives, computational capabilities, and communication resources.<n>In this paper, we first outline the basic concepts of heterogeneous federated learning.<n>We then summarize the research challenges in federated learning in terms of five aspects: data, model, task, device, and communication.
  arXiv  Detail & Related papers  (2024-05-16T06:35:42Z)
• Continual Learning with Pre-Trained Models: A Survey [61.97613090666247]
  Continual Learning aims to overcome the catastrophic forgetting of former knowledge when learning new ones. This paper presents a comprehensive survey of the latest advancements in PTM-based CL.
  arXiv  Detail & Related papers  (2024-01-29T18:27:52Z)
• A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions [48.97008907275482]
  Clustering is a fundamental machine learning task which has been widely studied in the literature. Deep Clustering, i.e., jointly optimizing the representation learning and clustering, has been proposed and hence attracted growing attention in the community. We summarize the essential components of deep clustering and categorize existing methods by the ways they design interactions between deep representation learning and clustering.
  arXiv  Detail & Related papers  (2022-06-15T15:05:13Z)
• Enhancing Identification of Structure Function of Academic Articles Using Contextual Information [6.28532577139029]
  This paper takes articles of the ACL conference as the corpus to identify the structure function of academic articles. We employ the traditional machine learning models and deep learning models to construct the classifiers based on various feature input. Inspired by (2), this paper introduces contextual information into the deep learning models and achieved significant results.
  arXiv  Detail & Related papers  (2021-11-28T11:21:21Z)
• A Survey on Text Classification: From Shallow to Deep Learning [83.47804123133719]
  The last decade has seen a surge of research in this area due to the unprecedented success of deep learning. This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021. We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification.
  arXiv  Detail & Related papers  (2020-08-02T00:09:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.