FedAUXfdp: Differentially Private One-Shot Federated Distillation

• URL: http://arxiv.org/abs/2205.14960v1
• Date: Mon, 30 May 2022 09:57:33 GMT
• Title: FedAUXfdp: Differentially Private One-Shot Federated Distillation
• Abstract summary: FedAUX is an augmentation of federated distillation with robust results on even highly heterogeneous client data. In experiments with deep networks on large-scale image datasets, FedAUXfdp with strong differential privacy guarantees performs significantly better than other SOTA baselines.
• Score: 11.28272643576878
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning suffers in the case of non-iid local datasets, i.e., when the distributions of the clients' data are heterogeneous. One promising approach to this challenge is the recently proposed method FedAUX, an augmentation of federated distillation with robust results on even highly heterogeneous client data. FedAUX is a partially $(\epsilon, \delta)$-differentially private method, insofar as the clients' private data is protected in only part of the training it takes part in. This work contributes a fully differentially private extension, termed FedAUXfdp. In experiments with deep networks on large-scale image datasets, FedAUXfdp with strong differential privacy guarantees performs significantly better than other equally privatized SOTA baselines on non-iid client data in just a single communication round. Full privatization results in a negligible reduction in accuracy at all levels of data heterogeneity.

Related papers

• Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learning [16.981834481629885]
  One-shot federated learning (OSFL) addresses the communication overhead of federated learning by limiting training to a single round.<n>Recent work has addressed this challenge by aggregating client knowledge on the server through the construction of transferable synthetic datasets or distillates.<n>We propose FedKT-CSD (Federated Knowledge Transfer via Collaborative Synthetic Data), a framework inspired by neural image compression.
  arXiv  Detail & Related papers  (2026-07-08T15:56:58Z)
• Federated Learning With Individualized Privacy Through Client Sampling [2.0432201743624456]
  We propose an adapted method for enabling Individualized Differential Privacy (IDP) in Federated Learning (FL) We calculate client-specific sampling rates based on their heterogeneous privacy budgets and integrate them into a modified IDP-FedAvg algorithm. The experimental results demonstrate that our approach achieves clear improvements over uniform DP baselines, reducing the trade-off between privacy and utility.
  arXiv  Detail & Related papers  (2025-01-29T13:11:21Z)
• Federated Instruction Tuning of LLMs with Domain Coverage Augmentation [35.54111318340366]
  Federated Domain-specific Instruction Tuning (FedDIT) utilizes limited cross-client private data together with server-side public data for instruction augmentation. We propose FedDCA, which optimize domain coverage through greedy client center selection and retrieval-based augmentation. We also investigate privacy preservation against memory extraction attacks utilizing various amounts of public data.
  arXiv  Detail & Related papers  (2024-09-30T09:34:31Z)
• Federated Transfer Learning with Differential Privacy [21.50525027559563]
  Federated learning has emerged as a powerful framework for analysing distributed data. In this paper, we aim to enhance learning on a target data set by leveraging information from multiple heterogeneous source data sets. We rigorously formulate the notion of federated differential privacy, which offers privacy guarantees for each data set.
  arXiv  Detail & Related papers  (2024-03-17T21:04:48Z)
• Clients Collaborate: Flexible Differentially Private Federated Learning with Guaranteed Improvement of Utility-Privacy Trade-off [34.2117116062642]
  We introduce a novel federated learning framework with rigorous privacy guarantees, named FedCEO, to strike a trade-off between model utility and user privacy. We show that our FedCEO can effectively recover the disrupted semantic information by smoothing the global semantic space. It observes significant performance improvements and strict privacy guarantees under different privacy settings.
  arXiv  Detail & Related papers  (2024-02-10T17:39:34Z)
• Federated Learning Empowered by Generative Content [55.576885852501775]
  Federated learning (FL) enables leveraging distributed private data for model training in a privacy-preserving way. We propose a novel FL framework termed FedGC, designed to mitigate data heterogeneity issues by diversifying private data with generative content. We conduct a systematic empirical study on FedGC, covering diverse baselines, datasets, scenarios, and modalities.
  arXiv  Detail & Related papers  (2023-12-10T07:38:56Z)
• FLAIM: AIM-based Synthetic Data Generation in the Federated Setting [18.38046354606749]
  DistAIM and FLAIM are proposed to produce synthetic data that mirrors the statistical properties of private data. We show that naively federating AIM can lead to substantial degradation in utility under the presence of heterogeneity.
  arXiv  Detail & Related papers  (2023-10-05T10:34:47Z)
• Smooth Anonymity for Sparse Graphs [69.1048938123063]
  differential privacy has emerged as the gold standard of privacy, however, when it comes to sharing sparse datasets. In this work, we consider a variation of $k$-anonymity, which we call smooth-$k$-anonymity, and design simple large-scale algorithms that efficiently provide smooth-$k$-anonymity.
  arXiv  Detail & Related papers  (2022-07-13T17:09:25Z)
• Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent [69.14164921515949]
  We characterize privacy guarantees for individual examples when releasing models trained by DP-SGD. We find that most examples enjoy stronger privacy guarantees than the worst-case bound. This implies groups that are underserved in terms of model utility simultaneously experience weaker privacy guarantees.
  arXiv  Detail & Related papers  (2022-06-06T13:49:37Z)
• Federated Learning in Non-IID Settings Aided by Differentially Private Synthetic Data [20.757477553095637]
  Federated learning (FL) is a privacy-promoting framework that enables clients to collaboratively train machine learning models. A major challenge in federated learning arises when the local data is heterogeneous. We propose FedDPMS, an FL algorithm in which clients deploy variational auto-encoders to augment local datasets with data synthesized using differentially private means of latent data representations.
  arXiv  Detail & Related papers  (2022-06-01T18:00:48Z)
• Mixed Differential Privacy in Computer Vision [133.68363478737058]
  AdaMix is an adaptive differentially private algorithm for training deep neural network classifiers using both private and public image data. A few-shot or even zero-shot learning baseline that ignores private data can outperform fine-tuning on a large private dataset.
  arXiv  Detail & Related papers  (2022-03-22T06:15:43Z)
• FedMix: Approximation of Mixup under Mean Augmented Federated Learning [60.503258658382]
  Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device. Current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases. We propose a new augmentation algorithm, named FedMix, which is inspired by a phenomenal yet simple data augmentation method, Mixup.
  arXiv  Detail & Related papers  (2021-07-01T06:14:51Z)
• Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy [67.4471689755097]
  This paper empirically demonstrates that the clipped FedAvg can perform surprisingly well even with substantial data heterogeneity. We provide the convergence analysis of a differential private (DP) FedAvg algorithm and highlight the relationship between clipping bias and the distribution of the clients' updates.
  arXiv  Detail & Related papers  (2021-06-25T14:47:19Z)
• Federated $f$-Differential Privacy [19.499120576896228]
  Federated learning (FL) is a training paradigm where the clients collaboratively learn models by repeatedly sharing information. We introduce federated $f$-differential privacy, a new notion specifically tailored to the federated setting. We then propose a generic private federated learning framework PriFedSync that accommodates a large family of state-of-the-art FL algorithms.
  arXiv  Detail & Related papers  (2021-02-22T16:28:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.