HePCo: Data-Free Heterogeneous Prompt Consolidation for Continual
Federated Learning
- URL: http://arxiv.org/abs/2306.09970v1
- Date: Fri, 16 Jun 2023 17:02:12 GMT
- Title: HePCo: Data-Free Heterogeneous Prompt Consolidation for Continual
Federated Learning
- Authors: Shaunak Halbe, James Seale Smith, Junjiao Tian, Zsolt Kira
- Abstract summary: We focus on the important yet understudied problem of Continual Federated Learning (CFL)
CFL is where a server communicates with a set of clients to incrementally learn new concepts without sharing or storing any data.
We propose a novel and lightweight generation and distillation scheme to consolidate client models at the server.
- Score: 21.639199127980508
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this paper, we focus on the important yet understudied problem of
Continual Federated Learning (CFL), where a server communicates with a set of
clients to incrementally learn new concepts over time without sharing or
storing any data. The complexity of this problem is compounded by challenges
from both the Continual and Federated Learning perspectives. Specifically,
models trained in a CFL setup suffer from catastrophic forgetting which is
exacerbated by data heterogeneity across clients. Existing attempts at this
problem tend to impose large overheads on clients and communication channels or
require access to stored data which renders them unsuitable for real-world use
due to privacy. In this paper, we attempt to tackle forgetting and
heterogeneity while minimizing overhead costs and without requiring access to
any stored data. We achieve this by leveraging a prompting based approach (such
that only prompts and classifier heads have to be communicated) and proposing a
novel and lightweight generation and distillation scheme to consolidate client
models at the server. We formulate this problem for image classification and
establish strong baselines for comparison, conduct experiments on CIFAR-100 as
well as challenging, large-scale datasets like ImageNet-R and DomainNet. Our
approach outperforms both existing methods and our own baselines by as much as
7% while significantly reducing communication and client-level computation
costs.
Related papers
- Fair Federated Data Clustering through Personalization: Bridging the Gap between Diverse Data Distributions [2.7905216619150344]
We introduce the idea of personalization in federated clustering. The goal is achieve balance between achieving lower clustering cost and at same time achieving uniform cost across clients.
We propose p-FClus that addresses these goal in a single round of communication between server and clients.
arXiv Detail & Related papers (2024-07-05T07:10:26Z) - An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets.
Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round.
We introduce FedAF, a novel aggregation-free FL algorithm.
arXiv Detail & Related papers (2024-04-29T05:55:23Z) - A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated
Class Incremental Learning for Vision Tasks [34.971800168823215]
This paper presents a framework for $textbffederated class incremental learning$ that utilizes a generative model to synthesize samples from past distributions.
To preserve privacy, the generative model is trained on the server using data-free methods at the end of each task without requesting data from clients.
arXiv Detail & Related papers (2023-11-13T22:21:27Z) - Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training.
In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework.
Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
arXiv Detail & Related papers (2023-08-11T09:58:47Z) - Better Generative Replay for Continual Federated Learning [20.57194599280318]
Federated learning is a technique that enables a centralized server to learn from distributed clients via communications.
In this paper, we introduce the problem of continual federated learning, where clients incrementally learn new tasks and history data cannot be stored.
We propose our FedCIL model with two simple but effective solutions: model consolidation and consistency enforcement.
arXiv Detail & Related papers (2023-02-25T06:26:56Z) - Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device)
In FL, each data holder trains a model locally and releases it to a central server for aggregation.
In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation).
In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
arXiv Detail & Related papers (2022-11-20T10:49:22Z) - FedNet2Net: Saving Communication and Computations in Federated Learning
with Model Growing [0.0]
Federated learning (FL) is a recently developed area of machine learning.
In this paper, a novel scheme based on the notion of "model growing" is proposed.
The proposed approach is tested extensively on three standard benchmarks and is shown to achieve substantial reduction in communication and client computation.
arXiv Detail & Related papers (2022-07-19T21:54:53Z) - Federated Multi-Target Domain Adaptation [99.93375364579484]
Federated learning methods enable us to train machine learning models on distributed user data while preserving its privacy.
We consider a more practical scenario where the distributed client data is unlabeled, and a centralized labeled dataset is available on the server.
We propose an effective DualAdapt method to address the new challenges.
arXiv Detail & Related papers (2021-08-17T17:53:05Z) - Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data.
We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks.
We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
arXiv Detail & Related papers (2021-04-27T18:23:54Z) - Federated Learning with Taskonomy for Non-IID Data [0.0]
We introduce federated learning with taskonomy.
In a one-off process, the server provides the clients with a pretrained (and fine-tunable) encoder to compress their data into a latent representation, and transmit the signature of their data back to the server.
The server then learns the task-relatedness among clients via manifold learning, and performs a generalization of federated averaging.
arXiv Detail & Related papers (2021-03-29T20:47:45Z) - Federated Continual Learning with Weighted Inter-client Transfer [79.93004004545736]
We propose a novel federated continual learning framework, Federated Weighted Inter-client Transfer (FedWeIT)
FedWeIT decomposes the network weights into global federated parameters and sparse task-specific parameters, and each client receives selective knowledge from other clients.
We validate our FedWeIT against existing federated learning and continual learning methods, and our model significantly outperforms them with a large reduction in the communication cost.
arXiv Detail & Related papers (2020-03-06T13:33:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.