Distributed, communication-efficient, and differentially private estimation of KL divergence

• URL: http://arxiv.org/abs/2411.16478v1
• Date: Mon, 25 Nov 2024 15:20:40 GMT
• Title: Distributed, communication-efficient, and differentially private estimation of KL divergence
• Authors: Mary Scott, Sayan Biswas, Graham Cormode, Carsten Maple,
• Abstract summary: Key task in managing distributed, sensitive data is to measure the extent to which a distribution changes. We describe novel algorithmic approaches for estimating the KL divergence of data across federated models of computation, under differential privacy.
• Score: 15.294136011320433
• License:
• Abstract: A key task in managing distributed, sensitive data is to measure the extent to which a distribution changes. Understanding this drift can effectively support a variety of federated learning and analytics tasks. However, in many practical settings sharing such information can be undesirable (e.g., for privacy concerns) or infeasible (e.g., for high communication costs). In this work, we describe novel algorithmic approaches for estimating the KL divergence of data across federated models of computation, under differential privacy. We analyze their theoretical properties and present an empirical study of their performance. We explore parameter settings that optimize the accuracy of the algorithm catering to each of the settings; these provide sub-variations that are applicable to real-world tasks, addressing different context- and application-specific trust level requirements. Our experimental results confirm that our private estimators achieve accuracy comparable to a baseline algorithm without differential privacy guarantees.

Related papers

• Optimal Federated Learning for Nonparametric Regression with Heterogeneous Distributed Differential Privacy Constraints [5.3595271893779906]
  We study federated learning for nonparametric regression in the context of distributed samples across different servers. Findings shed light on the tradeoff between statistical accuracy and privacy preservation.
  arXiv  Detail & Related papers  (2024-06-10T19:34:07Z)
• Differentially Private Federated Learning: Servers Trustworthiness, Estimation, and Statistical Inference [18.97060758177909]
  This paper investigates the challenges of high-dimensional estimation and inference under the constraints of differential privacy. We introduce a novel federated estimation algorithm tailored for linear regression models. We also propose methods for statistical inference, including coordinate-wise confidence intervals for individual parameters.
  arXiv  Detail & Related papers  (2024-04-25T02:14:07Z)
• Personalized Graph Federated Learning with Differential Privacy [6.282767337715445]
  This paper presents a personalized graph federated learning (PGFL) framework in which distributedly connected servers and their respective edge devices collaboratively learn device or cluster-specific models. We study a variant of the PGFL implementation that utilizes differential privacy, specifically zero-concentrated differential privacy, where a noise sequence perturbs model exchanges. Our analysis shows that the algorithm ensures local differential privacy for all clients in terms of zero-concentrated differential privacy.
  arXiv  Detail & Related papers  (2023-06-10T09:52:01Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Private Domain Adaptation from a Public Source [48.83724068578305]
  We design differentially private discrepancy-based algorithms for adaptation from a source domain with public labeled data to a target domain with unlabeled private data. Our solutions are based on private variants of Frank-Wolfe and Mirror-Descent algorithms.
  arXiv  Detail & Related papers  (2022-08-12T06:52:55Z)
• Assaying Out-Of-Distribution Generalization in Transfer Learning [103.57862972967273]
  We take a unified view of previous work, highlighting message discrepancies that we address empirically. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting.
  arXiv  Detail & Related papers  (2022-07-19T12:52:33Z)
• DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixup [58.894901088797376]
  federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data. We propose a general framework to solve the above two challenges simultaneously. We provide comprehensive theoretical analysis including robustness analysis, convergence analysis, and generalization ability.
  arXiv  Detail & Related papers  (2022-04-16T08:08:29Z)
• Sensitivity analysis in differentially private machine learning using hybrid automatic differentiation [54.88777449903538]
  We introduce a novel textithybrid automatic differentiation (AD) system for sensitivity analysis. This enables modelling the sensitivity of arbitrary differentiable function compositions, such as the training of neural networks on private data. Our approach can enable the principled reasoning about privacy loss in the setting of data processing.
  arXiv  Detail & Related papers  (2021-07-09T07:19:23Z)
• Non-parametric Differentially Private Confidence Intervals for the Median [3.205141100055992]
  This paper proposes and evaluates several strategies to compute valid differentially private confidence intervals for the median. We also illustrate that addressing both sources of uncertainty--the error from sampling and the error from protecting the output--should be preferred over simpler approaches that incorporate the uncertainty in a sequential fashion.
  arXiv  Detail & Related papers  (2021-06-18T19:45:37Z)
• A Theoretical Perspective on Differentially Private Federated Multi-task Learning [12.935153199667987]
  collaborative learning models need to be developed with respect to both privacy and utility concerns. We propose a new federated multi-task for effective parameter transfer differential privacy to protect at the client level. We are the first to provide both privacy utility guarantees for such a proposed algorithm.
  arXiv  Detail & Related papers  (2020-11-14T00:53:16Z)
• Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach [150.8920602230832]
  We propose a framework for learning calibrated uncertainties under domain shifts. In particular, the density ratio estimation reflects the closeness of a target (test) sample to the source (training) distribution. We show that our proposed method generates calibrated uncertainties that benefit downstream tasks.
  arXiv  Detail & Related papers  (2020-10-08T02:10:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.