SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision

• URL: http://arxiv.org/abs/2110.02470v1
• Date: Wed, 6 Oct 2021 02:58:45 GMT
• Title: SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• Authors: Chaoyang He, Zhengyu Yang, Erum Mushtaq, Sunwoo Lee, Mahdi Soltanolkotabi, Salman Avestimehr
• Abstract summary: Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices. We propose self-supervised federated learning (SSFL), a unified self-supervised and personalized federated learning framework. We show that the gap of evaluation accuracy between supervised learning and unsupervised learning in FL is both small and reasonable.
• Score: 34.38856587032084
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated Learning (FL) is transforming the ML training ecosystem from a centralized over-the-cloud setting to distributed training over edge devices in order to strengthen data privacy. An essential but rarely studied challenge in FL is label deficiency at the edge. This problem is even more pronounced in FL compared to centralized training due to the fact that FL users are often reluctant to label their private data. Furthermore, due to the heterogeneous nature of the data at edge devices, it is crucial to develop personalized models. In this paper we propose self-supervised federated learning (SSFL), a unified self-supervised and personalized federated learning framework, and a series of algorithms under this framework which work towards addressing these challenges. First, under the SSFL framework, we demonstrate that the standard FedAvg algorithm is compatible with recent breakthroughs in centralized self-supervised learning such as SimSiam networks. Moreover, to deal with data heterogeneity at the edge devices in this framework, we have innovated a series of algorithms that broaden existing supervised personalization algorithms into the setting of self-supervised learning. We further propose a novel personalized federated self-supervised learning algorithm, Per-SSFL, which balances personalization and consensus by carefully regulating the distance between the local and global representations of data. To provide a comprehensive comparative analysis of all proposed algorithms, we also develop a distributed training system and related evaluation protocol for SSFL. Our findings show that the gap of evaluation accuracy between supervised learning and unsupervised learning in FL is both small and reasonable. The performance comparison indicates the representation regularization-based personalization method is able to outperform other variants.

Related papers

• Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneity [14.313847382199059]
  Federated quantization-based self-supervised learning scheme (Fed-QSSL) designed to address heterogeneity in FL systems. Fed-QSSL deploys de-quantization, weighted aggregation and re-quantization, ultimately creating models personalized to both data distribution and specific infrastructure of each client's device.
  arXiv  Detail & Related papers  (2023-12-20T19:11:19Z)
• Privacy-preserving Federated Primal-dual Learning for Non-convex and Non-smooth Problems with Model Sparsification [51.04894019092156]
  Federated learning (FL) has been recognized as a rapidly growing area, where the model is trained over clients under the FL orchestration (PS) In this paper, we propose a novel primal sparification algorithm for and guarantee non-smooth FL problems. Its unique insightful properties and its analyses are also presented.
  arXiv  Detail & Related papers  (2023-10-30T14:15:47Z)
• Efficient Cluster Selection for Personalized Federated Learning: A Multi-Armed Bandit Approach [2.5477011559292175]
  Federated learning (FL) offers a decentralized training approach for machine learning models, prioritizing data privacy. In this paper, we introduce a dynamic Upper Confidence Bound (dUCB) algorithm inspired by the multi-armed bandit (MAB) approach.
  arXiv  Detail & Related papers  (2023-10-29T16:46:50Z)
• Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
  We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
  arXiv  Detail & Related papers  (2023-07-12T16:28:21Z)
• Combating Exacerbated Heterogeneity for Robust Models in Federated Learning [91.88122934924435]
  Combination of adversarial training and federated learning can lead to the undesired robustness deterioration. We propose a novel framework called Slack Federated Adversarial Training (SFAT) We verify the rationality and effectiveness of SFAT on various benchmarked and real-world datasets.
  arXiv  Detail & Related papers  (2023-03-01T06:16:15Z)
• Heterogeneous Federated Learning via Grouped Sequential-to-Parallel Training [60.892342868936865]
  Federated learning (FL) is a rapidly growing privacy-preserving collaborative machine learning paradigm. We propose a data heterogeneous-robust FL approach, FedGSP, to address this challenge. We show that FedGSP improves the accuracy by 3.7% on average compared with seven state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-01-31T03:15:28Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)
• Communication-Efficient Hierarchical Federated Learning for IoT Heterogeneous Systems with Imbalanced Data [42.26599494940002]
  Federated learning (FL) is a distributed learning methodology that allows multiple nodes to cooperatively train a deep learning model. This paper studies the potential of hierarchical FL in IoT heterogeneous systems. It proposes an optimized solution for user assignment and resource allocation on multiple edge nodes.
  arXiv  Detail & Related papers  (2021-07-14T08:32:39Z)
• FedMix: Approximation of Mixup under Mean Augmented Federated Learning [60.503258658382]
  Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device. Current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases. We propose a new augmentation algorithm, named FedMix, which is inspired by a phenomenal yet simple data augmentation method, Mixup.
  arXiv  Detail & Related papers  (2021-07-01T06:14:51Z)
• FedSemi: An Adaptive Federated Semi-Supervised Learning Framework [23.90642104477983]
  Federated learning (FL) has emerged as an effective technique to co-training machine learning models without actually sharing data and leaking privacy. Most existing FL methods focus on the supervised setting and ignore the utilization of unlabeled data. We propose FedSemi, a novel, adaptive, and general framework, which firstly introduces the consistency regularization into FL using a teacher-student model.
  arXiv  Detail & Related papers  (2020-12-06T15:46:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.