A Survey on Federated Fine-tuning of Large Language Models

• URL: http://arxiv.org/abs/2503.12016v1
• Date: Sat, 15 Mar 2025 06:52:10 GMT
• Title: A Survey on Federated Fine-tuning of Large Language Models
• Authors: Yebo Wu, Chunlin Tian, Jingguang Li, He Sun, Kahou Tam, Li Li, Chengzhong Xu,
• Abstract summary: Federated Learning (FL) offers a promising approach that enables collaborative model adaptation while ensuring data privacy.<n>We first trace the historical evolution of both Large Language Models (LLMs) and FL, while summarizing relevant prior surveys.<n>Following this, we conduct an extensive study of existing parameter-efficient fine-tuning (PEFT) methods and explore their applicability in FL.<n>Finally, we identify critical open challenges and outline promising research directions to drive future advancements in FedLLM.
• Score: 17.79395946441051
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks, with fine-tuning playing a pivotal role in adapting them to specific downstream applications. Federated Learning (FL) offers a promising approach that enables collaborative model adaptation while ensuring data privacy, i.e., FedLLM. In this survey, we provide a systematic and thorough review of the integration of LLMs with FL. Specifically, we first trace the historical evolution of both LLMs and FL, while summarizing relevant prior surveys. We then present an in-depth analysis of the fundamental challenges encountered in deploying FedLLM. Following this, we conduct an extensive study of existing parameter-efficient fine-tuning (PEFT) methods and explore their applicability in FL. Furthermore, we introduce a comprehensive evaluation benchmark to rigorously assess FedLLM performance and discuss its diverse real-world applications across multiple domains. Finally, we identify critical open challenges and outline promising research directions to drive future advancements in FedLLM. We maintain an active \href{https://github.com/Clin0212/Awesome-Federated-LLM-Learning}{GitHub repository} tracking cutting-edge advancements. This survey serves as a foundational resource for researchers and practitioners, offering insights into the evolving landscape of federated fine-tuning for LLMs while guiding future innovations in privacy-preserving AI.

Related papers

• LLM Post-Training: A Deep Dive into Reasoning Large Language Models [131.10969986056]
  Large Language Models (LLMs) have transformed the natural language processing landscape and brought to life diverse applications. Post-training methods enable LLMs to refine their knowledge, improve reasoning, enhance factual accuracy, and align more effectively with user intents and ethical considerations.
  arXiv  Detail & Related papers  (2025-02-28T18:59:54Z)
• A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models [36.601209595620446]
  This study investigates the machine unlearning techniques within the context of large language models (LLMs)<n>LLMs unlearning offers a principled approach to removing the influence of undesirable data from LLMs.<n>Despite growing research interest, there is no comprehensive survey that systematically organizes existing work and distills key insights.
  arXiv  Detail & Related papers  (2025-02-22T12:46:14Z)
• Federated Large Language Models: Current Progress and Future Directions [63.68614548512534]
  This paper surveys Federated learning for LLMs (FedLLM), highlighting recent advances and future directions. We focus on two key aspects: fine-tuning and prompt learning in a federated setting, discussing existing work and associated research challenges.
  arXiv  Detail & Related papers  (2024-09-24T04:14:33Z)
• Vertical Federated Learning for Effectiveness, Security, Applicability: A Survey [67.48187503803847]
  Vertical Federated Learning (VFL) is a privacy-preserving distributed learning paradigm. Recent research has shown promising results addressing various challenges in VFL. This survey offers a systematic overview of recent developments.
  arXiv  Detail & Related papers  (2024-05-25T16:05:06Z)
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [70.38817963253034]
  This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution. We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
  arXiv  Detail & Related papers  (2023-09-01T09:40:36Z)
• Integration of Large Language Models and Federated Learning [58.9876604258949]
  We propose a research framework, dividing the fusion of LLMs and FL into three parts. We first provide a review of the current state of research in the domain of LLMs combined with FL, including their typical applications. We then discuss the practical applications of the combination of LLMs and FL in critical scenarios such as healthcare, finance, and education.
  arXiv  Detail & Related papers  (2023-07-18T02:09:14Z)
• A Survey on Large Language Models for Recommendation [77.91673633328148]
  Large Language Models (LLMs) have emerged as powerful tools in the field of Natural Language Processing (NLP) This survey presents a taxonomy that categorizes these models into two major paradigms, respectively Discriminative LLM for Recommendation (DLLM4Rec) and Generative LLM for Recommendation (GLLM4Rec)
  arXiv  Detail & Related papers  (2023-05-31T13:51:26Z)
• Towards Interpretable Federated Learning [19.764172768506132]
  Federated learning (FL) enables multiple data owners to build machine learning models collaboratively without exposing their private local data. It is important to balance the need for performance, privacy-preservation and interpretability, especially in mission critical applications such as finance and healthcare. We conduct comprehensive analysis of the representative IFL approaches, the commonly adopted performance evaluation metrics, and promising directions towards building versatile IFL techniques.
  arXiv  Detail & Related papers  (2023-02-27T02:06:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.