Decoupled Federated Learning for ASR with Non-IID Data
- URL: http://arxiv.org/abs/2206.09102v1
- Date: Sat, 18 Jun 2022 03:44:37 GMT
- Title: Decoupled Federated Learning for ASR with Non-IID Data
- Authors: Han Zhu, Jindong Wang, Gaofeng Cheng, Pengyuan Zhang, Yonghong Yan
- Abstract summary: We tackle the non-IID issue in FL-based ASR with personalized FL, which learns personalized models for each client.
Experiments demonstrate two proposed personalized FL-based ASR approaches could reduce WER by 2.3% - 3.4% compared with FedAvg.
- Score: 34.59790627669783
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Automatic speech recognition (ASR) with federated learning (FL) makes it
possible to leverage data from multiple clients without compromising privacy.
The quality of FL-based ASR could be measured by recognition performance,
communication and computation costs. When data among different clients are not
independently and identically distributed (non-IID), the performance could
degrade significantly. In this work, we tackle the non-IID issue in FL-based
ASR with personalized FL, which learns personalized models for each client.
Concretely, we propose two types of personalized FL approaches for ASR.
Firstly, we adapt the personalization layer based FL for ASR, which keeps some
layers locally to learn personalization models. Secondly, to reduce the
communication and computation costs, we propose decoupled federated learning
(DecoupleFL). On one hand, DecoupleFL moves the computation burden to the
server, thus decreasing the computation on clients. On the other hand,
DecoupleFL communicates secure high-level features instead of model parameters,
thus reducing communication cost when models are large. Experiments demonstrate
two proposed personalized FL-based ASR approaches could reduce WER by 2.3% -
3.4% compared with FedAvg. Among them, DecoupleFL has only 11.4% communication
and 75% computation cost compared with FedAvg, which is also significantly less
than the personalization layer based FL.
Related papers
- Revisiting Ensembling in One-Shot Federated Learning [9.02411690527967]
One-shot federated learning (OFL) trades the iterative exchange of models between clients and the server with a single round of communication.
We introduce FENS, a novel federated ensembling scheme that approaches the accuracy of FL with the communication efficiency of OFL.
FENS achieves up to a 26.9% higher accuracy over state-of-the-art (SOTA) OFL, being only 3.1% lower than FL.
arXiv Detail & Related papers (2024-11-11T17:58:28Z) - Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training.
In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework.
Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
arXiv Detail & Related papers (2023-08-11T09:58:47Z) - User-Centric Federated Learning: Trading off Wireless Resources for
Personalization [18.38078866145659]
In Federated Learning (FL) systems, Statistical Heterogeneousness increases the algorithm convergence time and reduces the generalization performance.
To tackle the above problems without violating the privacy constraints that FL imposes, personalized FL methods have to couple statistically similar clients without directly accessing their data.
In this work, we design user-centric aggregation rules that are based on readily available gradient information and are capable of producing personalized models for each FL client.
Our algorithm outperforms popular personalized FL baselines in terms of average accuracy, worst node performance, and training communication overhead.
arXiv Detail & Related papers (2023-04-25T15:45:37Z) - Federated Learning of Shareable Bases for Personalization-Friendly Image
Classification [54.72892987840267]
FedBasis learns a set of few shareable basis'' models, which can be linearly combined to form personalized models for clients.
Specifically for a new client, only a small set of combination coefficients, not the model weights, needs to be learned.
To demonstrate the effectiveness and applicability of FedBasis, we also present a more practical PFL testbed for image classification.
arXiv Detail & Related papers (2023-04-16T20:19:18Z) - Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device)
In FL, each data holder trains a model locally and releases it to a central server for aggregation.
In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation).
In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
arXiv Detail & Related papers (2022-11-20T10:49:22Z) - Sparse Federated Learning with Hierarchical Personalized Models [24.763028713043468]
Federated learning (FL) can achieve privacy-safe and reliable collaborative training without collecting users' private data.
We propose a personalized FL algorithm using a hierarchical proximal mapping based on the moreau envelop, named sparse federated learning with hierarchical personalized models (sFedHP)
A continuously differentiable approximated L1-norm is also used as the sparse constraint to reduce the communication cost.
arXiv Detail & Related papers (2022-03-25T09:06:42Z) - Acceleration of Federated Learning with Alleviated Forgetting in Local
Training [61.231021417674235]
Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy.
We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage.
Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
arXiv Detail & Related papers (2022-03-05T02:31:32Z) - Achieving Personalized Federated Learning with Sparse Local Models [75.76854544460981]
Federated learning (FL) is vulnerable to heterogeneously distributed data.
To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user.
Existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory.
We proposeFedSpa, a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge.
arXiv Detail & Related papers (2022-01-27T08:43:11Z) - Personalized Federated Learning for Heterogeneous Clients with Clustered
Knowledge Transfer [25.041895804188748]
We propose PerFed-CKT, where clients can use heterogeneous model and do not directly communicate their model parameters.
We show that PerFed-CKT achieves high test accuracy with several orders of magnitude lower communication cost compared to the state-of-the-art personalized FL schemes.
arXiv Detail & Related papers (2021-09-16T17:13:53Z) - Dynamic Attention-based Communication-Efficient Federated Learning [85.18941440826309]
Federated learning (FL) offers a solution to train a global machine learning model.
FL suffers performance degradation when client data distribution is non-IID.
We propose a new adaptive training algorithm $textttAdaFL$ to combat this degradation.
arXiv Detail & Related papers (2021-08-12T14:18:05Z) - Personalized Federated Learning via Maximizing Correlation with Sparse
and Hierarchical Extensions [14.862798952297105]
Federated Learning (FL) is a collaborative machine learning technique to train a global model without obtaining clients' private data.
We propose a novel personalized federated learning via maximizing correlation pFedMac.
We show that pFedMac performs better than the L2-norm distance based personalization methods.
arXiv Detail & Related papers (2021-07-12T11:43:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.