FedClassAvg: Local Representation Learning for Personalized Federated
Learning on Heterogeneous Neural Networks
- URL: http://arxiv.org/abs/2210.14226v2
- Date: Thu, 27 Oct 2022 03:19:30 GMT
- Title: FedClassAvg: Local Representation Learning for Personalized Federated
Learning on Heterogeneous Neural Networks
- Authors: Jaehee Jang, Heonseok Ha, Dahuin Jung, Sungroh Yoon
- Abstract summary: We propose a novel personalized federated learning method called federated classifier averaging (FedClassAvg)
FedClassAvg aggregates weights as an agreement on decision boundaries on feature spaces.
We demonstrate it outperforms the current state-of-the-art algorithms on heterogeneous personalized federated learning tasks.
- Score: 21.613436984547917
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Personalized federated learning is aimed at allowing numerous clients to
train personalized models while participating in collaborative training in a
communication-efficient manner without exchanging private data. However, many
personalized federated learning algorithms assume that clients have the same
neural network architecture, and those for heterogeneous models remain
understudied. In this study, we propose a novel personalized federated learning
method called federated classifier averaging (FedClassAvg). Deep neural
networks for supervised learning tasks consist of feature extractor and
classifier layers. FedClassAvg aggregates classifier weights as an agreement on
decision boundaries on feature spaces so that clients with not independently
and identically distributed (non-iid) data can learn about scarce labels. In
addition, local feature representation learning is applied to stabilize the
decision boundaries and improve the local feature extraction capabilities for
clients. While the existing methods require the collection of auxiliary data or
model weights to generate a counterpart, FedClassAvg only requires clients to
communicate with a couple of fully connected layers, which is highly
communication-efficient. Moreover, FedClassAvg does not require extra
optimization problems such as knowledge transfer, which requires intensive
computation overhead. We evaluated FedClassAvg through extensive experiments
and demonstrated it outperforms the current state-of-the-art algorithms on
heterogeneous personalized federated learning tasks.
Related papers
- Personalized federated learning based on feature fusion [2.943623084019036]
Federated learning enables distributed clients to collaborate on training while storing their data locally to protect client privacy.
We propose a personalized federated learning approach called pFedPM.
In our process, we replace traditional gradient uploading with feature uploading, which helps reduce communication costs and allows for heterogeneous client models.
arXiv Detail & Related papers (2024-06-24T12:16:51Z) - Federated Face Forgery Detection Learning with Personalized Representation [63.90408023506508]
Deep generator technology can produce high-quality fake videos that are indistinguishable, posing a serious social threat.
Traditional forgery detection methods directly centralized training on data.
The paper proposes a novel federated face forgery detection learning with personalized representation.
arXiv Detail & Related papers (2024-06-17T02:20:30Z) - Personalized Federated Learning via Sequential Layer Expansion in Representation Learning [0.0]
Federated learning ensures the privacy of clients by conducting distributed training on individual client devices and sharing only the model weights with a central server.
We propose a new representation learning-based approach that suggests decoupling the entire deep learning model into more densely divided parts with the application of suitable scheduling methods.
arXiv Detail & Related papers (2024-04-27T06:37:19Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - Personalized Federated Learning with Feature Alignment and Classifier
Collaboration [13.320381377599245]
Data heterogeneity is one of the most challenging issues in federated learning.
One such approach in deep neural networks based tasks is employing a shared feature representation and learning a customized classifier head for each client.
In this work, we conduct explicit local-global feature alignment by leveraging global semantic knowledge for learning a better representation.
arXiv Detail & Related papers (2023-06-20T19:58:58Z) - Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner.
Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server.
This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
arXiv Detail & Related papers (2023-02-24T08:41:19Z) - Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device)
In FL, each data holder trains a model locally and releases it to a central server for aggregation.
In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation).
In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
arXiv Detail & Related papers (2022-11-20T10:49:22Z) - Architecture Agnostic Federated Learning for Neural Networks [19.813602191888837]
This work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN) framework.
FedHeNN allows each client to build a personalised model without enforcing a common architecture across clients.
The key idea of FedHeNN is to use the instance-level representations obtained from peer clients to guide the simultaneous training on each client.
arXiv Detail & Related papers (2022-02-15T22:16:06Z) - An Expectation-Maximization Perspective on Federated Learning [75.67515842938299]
Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device.
In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters.
We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
arXiv Detail & Related papers (2021-11-19T12:58:59Z) - Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client.
Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation.
This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
arXiv Detail & Related papers (2021-02-14T05:36:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.