Grounding Foundation Models through Federated Transfer Learning: A General Framework

• URL: http://arxiv.org/abs/2311.17431v9
• Date: Tue, 6 Feb 2024 12:18:29 GMT
• Title: Grounding Foundation Models through Federated Transfer Learning: A General Framework
• Authors: Yan Kang, Tao Fan, Hanlin Gu, Xiaojin Zhang, Lixin Fan, Qiang Yang
• Abstract summary: Foundation Models (FMs) such as GPT-4 have achieved remarkable success in various natural language processing and computer vision tasks. Grounding FMs by adapting them to domain-specific tasks or augmenting them with domain-specific knowledge enables us to exploit the full potential of FMs. In recent years, the need for grounding FMs leveraging Federated Transfer Learning (FTL) has arisen strongly in both academia and industry. Motivated by the strong growth in FTL-FM research and the potential impact of FTL-FM on industrial applications, we propose an FTL-FM framework that formulates problems of
• Score: 20.341440265217496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation Models (FMs) such as GPT-4 encoded with vast knowledge and powerful emergent abilities have achieved remarkable success in various natural language processing and computer vision tasks. Grounding FMs by adapting them to domain-specific tasks or augmenting them with domain-specific knowledge enables us to exploit the full potential of FMs. However, grounding FMs faces several challenges, stemming primarily from constrained computing resources, data privacy, model heterogeneity, and model ownership. Federated Transfer Learning (FTL), the combination of federated learning and transfer learning, provides promising solutions to address these challenges. In recent years, the need for grounding FMs leveraging FTL, coined FTL-FM, has arisen strongly in both academia and industry. Motivated by the strong growth in FTL-FM research and the potential impact of FTL-FM on industrial applications, we propose an FTL-FM framework that formulates problems of grounding FMs in the federated learning setting, construct a detailed taxonomy based on the FTL-FM framework to categorize state-of-the-art FTL-FM works, and comprehensively overview FTL-FM works based on the proposed taxonomy. We also establish correspondences between FTL-FM and conventional phases of adapting FM so that FM practitioners can align their research works with FTL-FM. In addition, we overview advanced efficiency-improving and privacy-preserving techniques because efficiency and privacy are critical concerns in FTL-FM. Last, we discuss opportunities and future research directions of FTL-FM.

Related papers

• Synergizing Foundation Models and Federated Learning: A Survey [23.416321895575507]
  This paper discusses the potentials and challenges of synergizing Federated Learning (FL) and Foundation Models (FM) FL is a collaborative learning paradigm that breaks the barrier of data availability from different participants. It provides a promising solution to customize and adapt FMs to a wide range of domain-specific tasks using distributed datasets whilst preserving privacy.
  arXiv  Detail & Related papers  (2024-06-18T17:58:09Z)
• FedPFT: Federated Proxy Fine-Tuning of Foundation Models [55.58899993272904]
  Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges as a promising strategy for protecting data privacy and valuable FMs. Existing methods fine-tune FM by allocating sub-FM to clients in FL, leading to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients. We propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation in downstream tasks through FL by two key modules.
  arXiv  Detail & Related papers  (2024-04-17T16:30:06Z)
• A Survey on Efficient Federated Learning Methods for Foundation Model Training [62.473245910234304]
  Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications.
  arXiv  Detail & Related papers  (2024-01-09T10:22:23Z)
• Learn From Model Beyond Fine-Tuning: A Survey [78.80920533793595]
  Learn From Model (LFM) focuses on the research, modification, and design of foundation models (FM) based on the model interface. The study of LFM techniques can be broadly categorized into five major areas: model tuning, model distillation, model reuse, meta learning and model editing. This paper gives a comprehensive review of the current methods based on FM from the perspective of LFM.
  arXiv  Detail & Related papers  (2023-10-12T10:20:36Z)
• The Role of Federated Learning in a Wireless World with Foundation Models [59.8129893837421]
  Foundation models (FMs) are general-purpose artificial intelligence (AI) models that have recently enabled multiple brand-new generative AI applications. Currently, the exploration of the interplay between FMs and federated learning (FL) is still in its nascent stage. This article explores the extent to which FMs are suitable for FL over wireless networks, including a broad overview of research challenges and opportunities.
  arXiv  Detail & Related papers  (2023-10-06T04:13:10Z)
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [70.38817963253034]
  This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution. We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
  arXiv  Detail & Related papers  (2023-09-01T09:40:36Z)
• When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions [47.00147534252281]
  The intersection of the Foundation Model (FM) and Federated Learning (FL) provides mutual benefits. FL expands the availability of data for FMs and enables computation sharing, distributing the training process and reducing the burden on FL participants. On the other hand, FM, with its enormous size, pre-trained knowledge, and exceptional performance, serves as a robust starting point for FL.
  arXiv  Detail & Related papers  (2023-06-27T15:15:55Z)
• Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models [8.184714897613166]
  We propose the Federated Foundation Models (FFMs) paradigm, which combines the benefits of FMs and Federated Learning (FL) We discuss the potential benefits and challenges of integrating FL into the lifespan of FMs, covering pre-training, fine-tuning, and application. We explore the possibility of continual/lifelong learning in FFMs, as increased computational power at the edge may unlock the potential for optimizing FMs using newly generated private data close to the data source.
  arXiv  Detail & Related papers  (2023-05-19T03:51:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.