Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher

• URL: http://arxiv.org/abs/2304.01731v4
• Date: Fri, 15 Dec 2023 03:21:09 GMT
• Title: Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher
• Authors: Jiawei Shao, Fangzhao Wu, Jun Zhang
• Abstract summary: Federated learning is vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. This paper proposes a selective knowledge sharing mechanism for FD, termed Selective-FD.
• Score: 52.2926020848095
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While federated learning is promising for privacy-preserving collaborative learning without revealing local data, it remains vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. Federated distillation (FD), built upon knowledge distillation--an effective technique for transferring knowledge from a teacher model to student models--emerges as an alternative paradigm, which provides enhanced privacy guarantees and addresses model heterogeneity. Nevertheless, challenges arise due to variations in local data distributions and the absence of a well-trained teacher model, which leads to misleading and ambiguous knowledge sharing that significantly degrades model performance. To address these issues, this paper proposes a selective knowledge sharing mechanism for FD, termed Selective-FD. It includes client-side selectors and a server-side selector to accurately and precisely identify knowledge from local and ensemble predictions, respectively. Empirical studies, backed by theoretical insights, demonstrate that our approach enhances the generalization capabilities of the FD framework and consistently outperforms baseline methods.

Related papers

• FEDLAD: Federated Evaluation of Deep Leakage Attacks and Defenses [50.921333548391345]
  Federated Learning is a privacy preserving decentralized machine learning paradigm. Recent research has revealed that private ground truth data can be recovered through a gradient technique known as Deep Leakage. This paper introduces the FEDLAD Framework (Federated Evaluation of Deep Leakage Attacks and Defenses), a comprehensive benchmark for evaluating Deep Leakage attacks and defenses.
  arXiv  Detail & Related papers  (2024-11-05T11:42:26Z)
• Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generator [19.00239208095762]
  Federated Learning (FL) is gaining popularity as a distributed learning framework that only shares model parameters or updates and keeps private data locally. We propose FedMD-CG, a novel FL method with highly competitive performance and high-level privacy preservation. We conduct extensive experiments on various image classification tasks to validate the superiority of FedMD-CG.
  arXiv  Detail & Related papers  (2024-09-11T02:36:36Z)
• Auto-selected Knowledge Adapters for Lifelong Person Re-identification [54.42307214981537]
  Lifelong Person Re-Identification requires systems to continually learn from non-overlapping datasets across different times and locations. Existing approaches, either rehearsal-free or rehearsal-based, still suffer from the problem of catastrophic forgetting. We introduce a novel framework AdalReID, that adopts knowledge adapters and a parameter-free auto-selection mechanism for lifelong learning.
  arXiv  Detail & Related papers  (2024-05-29T11:42:02Z)
• KnFu: Effective Knowledge Fusion [5.305607095162403]
  Federated Learning (FL) has emerged as a prominent alternative to the traditional centralized learning approach. The paper proposes Effective Knowledge Fusion (KnFu) algorithm that evaluates knowledge of local models to only fuse semantic neighbors' effective knowledge for each client. A key conclusion of the work is that in scenarios with large and highly heterogeneous local datasets, local training could be preferable to knowledge fusion-based solutions.
  arXiv  Detail & Related papers  (2024-03-18T15:49:48Z)
• Combating Exacerbated Heterogeneity for Robust Models in Federated Learning [91.88122934924435]
  Combination of adversarial training and federated learning can lead to the undesired robustness deterioration. We propose a novel framework called Slack Federated Adversarial Training (SFAT) We verify the rationality and effectiveness of SFAT on various benchmarked and real-world datasets.
  arXiv  Detail & Related papers  (2023-03-01T06:16:15Z)
• Feature Correlation-guided Knowledge Transfer for Federated Self-supervised Learning [19.505644178449046]
  We propose a novel and general method named Federated Self-supervised Learning with Feature-correlation based Aggregation (FedFoA) Our insight is to utilize feature correlation to align the feature mappings and calibrate the local model updates across clients during their local training process. We prove that FedFoA is a model-agnostic training framework and can be easily compatible with state-of-the-art unsupervised FL methods.
  arXiv  Detail & Related papers  (2022-11-14T13:59:50Z)
• Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation [20.24005399782197]
  Federated learning is a privacy-preserving machine learning paradigm. Recent proxy-data-free FD approaches can eliminate the need for additional public data, but suffer from remarkable discrepancy among local knowledge. We propose a proxy-data-free FD algorithm based on distributed knowledge congruence (FedDKC)
  arXiv  Detail & Related papers  (2022-04-14T15:39:22Z)
• Data-Free Knowledge Transfer: A Survey [13.335198869928167]
  knowledge distillation (KD) and domain adaptation (DA) are proposed and become research highlights. They both aim to transfer useful information from a well-trained model with original training data. Recently, the data-free knowledge transfer paradigm has attracted appealing attention.
  arXiv  Detail & Related papers  (2021-12-31T03:39:42Z)
• Transfer Heterogeneous Knowledge Among Peer-to-Peer Teammates: A Model Distillation Approach [55.83558520598304]
  We propose a brand new solution to reuse experiences and transfer value functions among multiple students via model distillation. We also describe how to design an efficient communication protocol to exploit heterogeneous knowledge. Our proposed framework, namely Learning and Teaching Categorical Reinforcement, shows promising performance on stabilizing and accelerating learning progress.
  arXiv  Detail & Related papers  (2020-02-06T11:31:04Z)
• Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification [106.08067870620218]
  We propose a self-paced knowledge distillation framework, termed Learning From Multiple Experts (LFME) We refer to these models as 'Experts', and the proposed LFME framework aggregates the knowledge from multiple 'Experts' to learn a unified student model. We conduct extensive experiments and demonstrate that our method is able to achieve superior performances compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-01-06T12:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.