Integration of Large Language Models and Federated Learning

• URL: http://arxiv.org/abs/2307.08925v3
• Date: Wed, 30 Oct 2024 03:04:21 GMT
• Title: Integration of Large Language Models and Federated Learning
• Authors: Chaochao Chen, Xiaohua Feng, Yuyuan Li, Lingjuan Lyu, Jun Zhou, Xiaolin Zheng, Jianwei Yin,
• Abstract summary: We propose a research framework, dividing the fusion of LLMs and FL into three parts. We first provide a review of the current state of research in the domain of LLMs combined with FL, including their typical applications. We then discuss the practical applications of the combination of LLMs and FL in critical scenarios such as healthcare, finance, and education.
• Score: 58.9876604258949
• License:
• Abstract: As the parameter size of Large Language Models (LLMs) continues to expand, there is an urgent need to address the scarcity of high-quality data. In response, existing research has attempted to make a breakthrough by incorporating Federated Learning (FL) into LLMs. Conversely, considering the outstanding performance of LLMs in task generalization, researchers have also tried applying LLMs within FL to tackle challenges in relevant domains. The complementarity between LLMs and FL has already ignited widespread research interest. In this paper, we aim to deeply explore the integration of LLMs and FL. We propose a research framework, dividing the fusion of LLMs and FL into three parts: the combination of LLM sub-technologies with FL, the integration of FL sub-technologies with LLMs, and the overall merger of LLMs and FL. We first provide a comprehensive review of the current state of research in the domain of LLMs combined with FL, including their typical applications, integration advantages, challenges faced, and future directions for resolution. Subsequently, we discuss the practical applications of the combination of LLMs and FL in critical scenarios such as healthcare, finance, and education, and provide new perspectives and insights into future research directions for LLMs and FL.

Related papers

• Federated Large Language Models: Current Progress and Future Directions [63.68614548512534]
  This paper surveys Federated learning for LLMs (FedLLM), highlighting recent advances and future directions. We focus on two key aspects: fine-tuning and prompt learning in a federated setting, discussing existing work and associated research challenges.
  arXiv  Detail & Related papers  (2024-09-24T04:14:33Z)
• Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement [72.97553348776425]
  We make a pioneering effort to broaden the applicability of merging techniques from FT to PT LLMs. We introduce an approach based on WeIght DisENtanglement (WIDEN) to effectively extend the merging scope. We merge Qwen1.5-Chat (an FT LLM with instruction-following skills) with Sailor (a PT LLM with multilingual abilities) across 7B and 14B model scales.
  arXiv  Detail & Related papers  (2024-08-06T10:46:46Z)
• FernUni LLM Experimental Infrastructure (FLEXI) -- Enabling Experimentation and Innovation in Higher Education Through Access to Open Large Language Models [2.190269031876989]
  We describe the current state of establishing an open LLM infrastructure at FernUniversit"at in Hagen under the project name FLEXI. The paper will provide some practical guidance for everyone trying to decide whether to run their own LLM server.
  arXiv  Detail & Related papers  (2024-06-27T09:46:11Z)
• Can LLMs Solve longer Math Word Problems Better? [47.227621867242]
  Math Word Problems (MWPs) are crucial for evaluating the capability of Large Language Models (LLMs) This study pioneers the exploration of Context Length Generalizability (CoLeG) Two novel metrics are proposed to assess the efficacy and resilience of LLMs in solving these problems.
  arXiv  Detail & Related papers  (2024-05-23T17:13:50Z)
• Large Language Models: A Survey [69.72787936480394]
  Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
  arXiv  Detail & Related papers  (2024-02-09T05:37:09Z)
• Knowledge Fusion of Large Language Models [73.28202188100646]
  This paper introduces the notion of knowledge fusion for large language models (LLMs) We externalize their collective knowledge and unique strengths, thereby elevating the capabilities of the target model beyond those of any individual source LLM. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation.
  arXiv  Detail & Related papers  (2024-01-19T05:02:46Z)
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [70.38817963253034]
  This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution. We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
  arXiv  Detail & Related papers  (2023-09-01T09:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.