Personalized Federated Continual Learning via Multi-granularity Prompt

• URL: http://arxiv.org/abs/2407.00113v1
• Date: Thu, 27 Jun 2024 13:41:37 GMT
• Title: Personalized Federated Continual Learning via Multi-granularity Prompt
• Authors: Hao Yu, Xin Yang, Xin Gao, Yan Kang, Hao Wang, Junbo Zhang, Tianrui Li,
• Abstract summary: We propose a novel concept called multi-granularity prompt, i.e., coarse-grained global prompt and fine-grained local prompt used to personalize the generalized representation. By the exclusive fusion of coarse-grained knowledge, we achieve the transmission and refinement of common knowledge among clients, further enhancing the performance of personalization.
• Score: 33.84680453375976
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Personalized Federated Continual Learning (PFCL) is a new practical scenario that poses greater challenges in sharing and personalizing knowledge. PFCL not only relies on knowledge fusion for server aggregation at the global spatial-temporal perspective but also needs model improvement for each client according to the local requirements. Existing methods, whether in Personalized Federated Learning (PFL) or Federated Continual Learning (FCL), have overlooked the multi-granularity representation of knowledge, which can be utilized to overcome Spatial-Temporal Catastrophic Forgetting (STCF) and adopt generalized knowledge to itself by coarse-to-fine human cognitive mechanisms. Moreover, it allows more effectively to personalized shared knowledge, thus serving its own purpose. To this end, we propose a novel concept called multi-granularity prompt, i.e., coarse-grained global prompt acquired through the common model learning process, and fine-grained local prompt used to personalize the generalized representation. The former focuses on efficiently transferring shared global knowledge without spatial forgetting, and the latter emphasizes specific learning of personalized local knowledge to overcome temporal forgetting. In addition, we design a selective prompt fusion mechanism for aggregating knowledge of global prompts distilled from different clients. By the exclusive fusion of coarse-grained knowledge, we achieve the transmission and refinement of common knowledge among clients, further enhancing the performance of personalization. Extensive experiments demonstrate the effectiveness of the proposed method in addressing STCF as well as improving personalized performance. Our code now is available at https://github.com/SkyOfBeginning/FedMGP.

Related papers

• Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models [7.810284483002312]
  We propose a novel framework that personalizes the prompt learning process through the lens of Mixture of Experts (MoE) pFedMoAP implements a local attention-based gating network that learns to generate enhanced text features for better alignment with local image data on the client. The results show that pFedMoAP consistently outperforms the state-of-the-art alternatives, underscoring its efficacy in personalizing prompt learning for CLIP.
  arXiv  Detail & Related papers  (2024-10-14T03:05:12Z)
• Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment [120.06538000214552]
  Adapting general large language models (LLMs) to specialized domains presents great challenges due to varied data distributions. We propose a new domain adaptation framework including domain knowledge learning and general format alignment, called Mix-CPT. Our proposed Mix-CPT framework can simultaneously improve the task-solving capabilities of LLMs on the target and general domains.
  arXiv  Detail & Related papers  (2024-07-15T15:20:13Z)
• Cross-Training with Multi-View Knowledge Fusion for Heterogenous Federated Learning [13.796783869133531]
  This paper presents a novel approach that enhances federated learning through a cross-training scheme incorporating multi-view information. Specifically, the proposed method, termed FedCT, includes three main modules, where the consistency-aware knowledge broadcasting module aims to optimize model assignment strategies. The multi-view knowledge-guided representation learning module leverages fused knowledge from both global and local views to enhance the preservation of local knowledge before and after model exchange. The mixup-based feature augmentation module aggregates rich information to further increase the diversity of feature spaces, which enables the model to better discriminate complex samples.
  arXiv  Detail & Related papers  (2024-05-30T13:27:30Z)
• A Unified and General Framework for Continual Learning [58.72671755989431]
  Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge. Various methods have been developed to address the challenge of catastrophic forgetting, including regularization-based, Bayesian-based, and memory-replay-based techniques. This research aims to bridge this gap by introducing a comprehensive and overarching framework that encompasses and reconciles these existing methodologies.
  arXiv  Detail & Related papers  (2024-03-20T02:21:44Z)
• KnFu: Effective Knowledge Fusion [5.305607095162403]
  Federated Learning (FL) has emerged as a prominent alternative to the traditional centralized learning approach. The paper proposes Effective Knowledge Fusion (KnFu) algorithm that evaluates knowledge of local models to only fuse semantic neighbors' effective knowledge for each client. A key conclusion of the work is that in scenarios with large and highly heterogeneous local datasets, local training could be preferable to knowledge fusion-based solutions.
  arXiv  Detail & Related papers  (2024-03-18T15:49:48Z)
• FedJETs: Efficient Just-In-Time Personalization with Federated Mixture of Experts [48.78037006856208]
  FedJETs is a novel solution by using a Mixture-of-Experts (MoE) framework within a Federated Learning (FL) setup. Our method leverages the diversity of the clients to train specialized experts on different subsets of classes, and a gating function to route the input to the most relevant expert(s) Our approach can improve accuracy up to 18% in state of the art FL settings, while maintaining competitive zero-shot performance.
  arXiv  Detail & Related papers  (2023-06-14T15:47:52Z)
• Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher [52.2926020848095]
  Federated learning is vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. This paper proposes a selective knowledge sharing mechanism for FD, termed Selective-FD.
  arXiv  Detail & Related papers  (2023-04-04T12:04:19Z)
• Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning [14.063276595895049]
  Federated Learning (FL) is a paradigm to train a global machine learning model across distributed clients without sharing raw data. We propose FedLU, a novel FL framework for heterogeneous KG embedding learning and unlearning. We show that FedLU achieves superior results in both link prediction and knowledge forgetting.
  arXiv  Detail & Related papers  (2023-02-04T02:44:48Z)
• Knowledge-Aware Federated Active Learning with Non-IID Data [75.98707107158175]
  We propose a federated active learning paradigm to efficiently learn a global model with limited annotation budget. The main challenge faced by federated active learning is the mismatch between the active sampling goal of the global model on the server and that of the local clients. We propose Knowledge-Aware Federated Active Learning (KAFAL), which consists of Knowledge-Specialized Active Sampling (KSAS) and Knowledge-Compensatory Federated Update (KCFU)
  arXiv  Detail & Related papers  (2022-11-24T13:08:43Z)
• Exploring Semantic Attributes from A Foundation Model for Federated Learning of Disjoint Label Spaces [46.59992662412557]
  In this work, we consider transferring mid-level semantic knowledge (such as attribute) which is not sensitive to specific objects of interest. We formulate a new Federated Zero-Shot Learning (FZSL) paradigm to learn mid-level semantic knowledge at multiple local clients. To improve model discriminative ability, we propose to explore semantic knowledge augmentation from external knowledge.
  arXiv  Detail & Related papers  (2022-08-29T10:05:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.