Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators

• URL: http://arxiv.org/abs/2307.05358v3
• Date: Mon, 11 Mar 2024 15:48:08 GMT
• Title: Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators
• Authors: Sikai Bai, Shuaicheng Li, Weiming Zhuang, Jie Zhang, Song Guo, Kunlin Yang, Jun Hou, Shuai Zhang, Junyu Gao, Shuai Yi
• Abstract summary: Federated semi-supervised learning (FSSL) emerges to train models from a small fraction of labeled data. We propose a novel FSSL framework with dual regulators, FedDure. We show that FedDure is superior to the existing methods across a wide range of settings.
• Score: 40.12377870379059
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning has become a popular method to learn from decentralized heterogeneous data. Federated semi-supervised learning (FSSL) emerges to train models from a small fraction of labeled data due to label scarcity on decentralized clients. Existing FSSL methods assume independent and identically distributed (IID) labeled data across clients and consistent class distribution between labeled and unlabeled data within a client. This work studies a more practical and challenging scenario of FSSL, where data distribution is different not only across clients but also within a client between labeled and unlabeled data. To address this challenge, we propose a novel FSSL framework with dual regulators, FedDure. FedDure lifts the previous assumption with a coarse-grained regulator (C-reg) and a fine-grained regulator (F-reg): C-reg regularizes the updating of the local model by tracking the learning effect on labeled data distribution; F-reg learns an adaptive weighting scheme tailored for unlabeled instances in each client. We further formulate the client model training as bi-level optimization that adaptively optimizes the model in the client with two regulators. Theoretically, we show the convergence guarantee of the dual regulators. Empirically, we demonstrate that FedDure is superior to the existing methods across a wide range of settings, notably by more than 11 on CIFAR-10 and CINIC-10 datasets.

Related papers

• (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning [4.803231218533992]
  Federated Learning (FL) is a distributed machine learning framework that trains accurate global models while preserving clients' privacy-sensitive data. Most FL approaches assume that clients possess labeled data, which is often not the case in practice. We propose $(FL)2$, a robust training method for unlabeled clients using sharpness-aware consistency regularization.
  arXiv  Detail & Related papers  (2024-10-30T17:15:02Z)
• FedClust: Tackling Data Heterogeneity in Federated Learning through Weight-Driven Client Clustering [26.478852701376294]
  Federated learning (FL) is an emerging distributed machine learning paradigm. One of the major challenges in FL is the presence of uneven data distributions across client devices. We propose em FedClust, a novel approach for CFL that leverages the correlation between local model weights and the data distribution of clients.
  arXiv  Detail & Related papers  (2024-07-09T02:47:16Z)
• Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
  Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training. In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework. Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
  arXiv  Detail & Related papers  (2023-08-11T09:58:47Z)
• Towards Unbiased Training in Federated Open-world Semi-supervised Learning [15.08153616709326]
  We propose a novel Federatedopen-world Semi-Supervised Learning (FedoSSL) framework, which can solve the key challenge in distributed and open-world settings. We adopt an uncertainty-aware suppressed loss to alleviate the biased training between locally unseen and globally unseen classes. The proposed FedoSSL can be easily adapted to state-of-the-art FL methods, which is also validated via extensive experiments on benchmarks and real-world datasets.
  arXiv  Detail & Related papers  (2023-05-01T11:12:37Z)
• Dual Class-Aware Contrastive Federated Semi-Supervised Learning [9.742389743497045]
  We present a novel Federated Semi-Supervised Learning (FSSL) method called Dual Class-aware Contrastive Federated Semi-Supervised Learning (DCCFSSL) By implementing a dual class-aware contrastive module, DCCFSSL establishes a unified training objective for different clients to tackle large deviations. Our experiments show that DCCFSSL outperforms current state-of-the-art methods on three benchmark datasets.
  arXiv  Detail & Related papers  (2022-11-16T13:54:31Z)
• Federated Learning from Only Unlabeled Data with Class-Conditional-Sharing Clients [98.22390453672499]
  Supervised federated learning (FL) enables multiple clients to share the trained model without sharing their labeled data. We propose federation of unsupervised learning (FedUL), where the unlabeled data are transformed into surrogate labeled data for each of the clients.
  arXiv  Detail & Related papers  (2022-04-07T09:12:00Z)
• Federated Semi-Supervised Learning with Class Distribution Mismatch [34.46190258291223]
  Federated semi-supervised learning (Fed-SSL) is an attractive solution for fully utilizing both labeled and unlabeled data. We introduce two proper regularization terms that can effectively alleviate the class distribution mismatch problem in Fed-SSL. We leverage the variance reduction and normalized averaging techniques to develop a novel Fed-SSL algorithm.
  arXiv  Detail & Related papers  (2021-10-29T14:18:20Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)
• Federated Unsupervised Representation Learning [56.715917111878106]
  We formulate a new problem in federated learning called Federated Unsupervised Representation Learning (FURL) to learn a common representation model without supervision. FedCA is composed of two key modules: dictionary module to aggregate the representations of samples from each client and share with all clients for consistency of representation space and alignment module to align the representation of each client on a base model trained on a public data.
  arXiv  Detail & Related papers  (2020-10-18T13:28:30Z)
• Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning [78.88007892742438]
  We study two essential scenarios of Federated Semi-Supervised Learning (FSSL) based on the location of the labeled data. We propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch)
  arXiv  Detail & Related papers  (2020-06-22T09:43:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.