Federated Semi-Supervised Learning with Prototypical Networks
- URL: http://arxiv.org/abs/2205.13921v2
- Date: Mon, 30 May 2022 15:31:09 GMT
- Title: Federated Semi-Supervised Learning with Prototypical Networks
- Authors: Woojung Kim, Keondo Park, Kihyuk Sohn, Raphael Shu, Hyung-Sin Kim
- Abstract summary: We propose ProtoFSSL, a novel FSSL approach based on prototypical networks.
In ProtoFSSL, clients share knowledge with each other via lightweight prototypes.
Compared to a FSSL approach based on weight sharing, the prototype-based inter-client knowledge sharing significantly reduces both communication and computation costs.
- Score: 18.82809442813657
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: With the increasing computing power of edge devices, Federated Learning (FL)
emerges to enable model training without privacy concerns. The majority of
existing studies assume the data are fully labeled on the client side. In
practice, however, the amount of labeled data is often limited. Recently,
federated semi-supervised learning (FSSL) is explored as a way to effectively
utilize unlabeled data during training. In this work, we propose ProtoFSSL, a
novel FSSL approach based on prototypical networks. In ProtoFSSL, clients share
knowledge with each other via lightweight prototypes, which prevents the local
models from diverging. For computing loss on unlabeled data, each client
creates accurate pseudo-labels based on shared prototypes. Jointly with labeled
data, the pseudo-labels provide training signals for local prototypes. Compared
to a FSSL approach based on weight sharing, the prototype-based inter-client
knowledge sharing significantly reduces both communication and computation
costs, enabling more frequent knowledge sharing between more clients for better
accuracy. In multiple datasets, ProtoFSSL results in higher accuracy compared
to the recent FSSL methods with and without knowledge sharing, such as
FixMatch, FedRGD, and FedMatch. On SVHN dataset, ProtoFSSL performs comparably
to fully supervised FL methods.
Related papers
- (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning [4.803231218533992]
Federated Learning (FL) is a distributed machine learning framework that trains accurate global models while preserving clients' privacy-sensitive data.
Most FL approaches assume that clients possess labeled data, which is often not the case in practice.
We propose $(FL)2$, a robust training method for unlabeled clients using sharpness-aware consistency regularization.
arXiv Detail & Related papers (2024-10-30T17:15:02Z) - Personalized federated learning based on feature fusion [2.943623084019036]
Federated learning enables distributed clients to collaborate on training while storing their data locally to protect client privacy.
We propose a personalized federated learning approach called pFedPM.
In our process, we replace traditional gradient uploading with feature uploading, which helps reduce communication costs and allows for heterogeneous client models.
arXiv Detail & Related papers (2024-06-24T12:16:51Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - ProtoCon: Pseudo-label Refinement via Online Clustering and Prototypical
Consistency for Efficient Semi-supervised Learning [60.57998388590556]
ProtoCon is a novel method for confidence-based pseudo-labeling.
Online nature of ProtoCon allows it to utilise the label history of the entire dataset in one training cycle.
It delivers significant gains and faster convergence over state-of-the-art datasets.
arXiv Detail & Related papers (2023-03-22T23:51:54Z) - Scalable Collaborative Learning via Representation Sharing [53.047460465980144]
Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device)
In FL, each data holder trains a model locally and releases it to a central server for aggregation.
In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation).
In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss.
arXiv Detail & Related papers (2022-11-20T10:49:22Z) - Dual Class-Aware Contrastive Federated Semi-Supervised Learning [9.742389743497045]
We present a novel Federated Semi-Supervised Learning (FSSL) method called Dual Class-aware Contrastive Federated Semi-Supervised Learning (DCCFSSL)
By implementing a dual class-aware contrastive module, DCCFSSL establishes a unified training objective for different clients to tackle large deviations.
Our experiments show that DCCFSSL outperforms current state-of-the-art methods on three benchmark datasets.
arXiv Detail & Related papers (2022-11-16T13:54:31Z) - Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data.
We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks.
We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
arXiv Detail & Related papers (2021-04-27T18:23:54Z) - WAFFLe: Weight Anonymized Factorization for Federated Learning [88.44939168851721]
In domains where data are sensitive or private, there is great value in methods that can learn in a distributed manner without the data ever leaving the local devices.
We propose Weight Anonymized Factorization for Federated Learning (WAFFLe), an approach that combines the Indian Buffet Process with a shared dictionary of weight factors for neural networks.
arXiv Detail & Related papers (2020-08-13T04:26:31Z) - Federated Semi-Supervised Learning with Inter-Client Consistency &
Disjoint Learning [78.88007892742438]
We study two essential scenarios of Federated Semi-Supervised Learning (FSSL) based on the location of the labeled data.
We propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch)
arXiv Detail & Related papers (2020-06-22T09:43:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.