Differentially Private Domain Adaptation with Theoretical Guarantees
- URL: http://arxiv.org/abs/2306.08838v2
- Date: Sun, 4 Feb 2024 22:00:31 GMT
- Title: Differentially Private Domain Adaptation with Theoretical Guarantees
- Authors: Raef Bassily, Corinna Cortes, Anqi Mao, Mehryar Mohri
- Abstract summary: In many applications, the labeled data at the labeled data's disposal is subject to privacy constraints and is relatively limited.
This is the modern problem of supervised domain adaptation from a public source to a private target domain.
We make use of a general learner to benefit from favorable theoretical learning guarantees.
- Score: 46.37771025567305
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In many applications, the labeled data at the learner's disposal is subject
to privacy constraints and is relatively limited. To derive a more accurate
predictor for the target domain, it is often beneficial to leverage publicly
available labeled data from an alternative domain, somewhat close to the target
domain. This is the modern problem of supervised domain adaptation from a
public source to a private target domain. We present two $(\epsilon,
\delta)$-differentially private adaptation algorithms for supervised
adaptation, for which we make use of a general optimization problem, recently
shown to benefit from favorable theoretical learning guarantees. Our first
algorithm is designed for regression with linear predictors and shown to solve
a convex optimization problem. Our second algorithm is a more general solution
for loss functions that may be non-convex but Lipschitz and smooth. While our
main objective is a theoretical analysis, we also report the results of several
experiments first demonstrating that the non-private versions of our algorithms
outperform adaptation baselines and next showing that, for larger values of the
target sample size or $\epsilon$, the performance of our private algorithms
remains close to that of the non-private formulation.
Related papers
- Generating Private Synthetic Data with Genetic Algorithms [29.756119782419955]
We study the problem of efficiently generating differentially private synthetic data that approximates the statistical properties of an underlying sensitive dataset.
We propose Private-GSD, a private genetic algorithm based on zeroth-order optimizations that do not require modifying the original objective.
We show that Private-GSD outperforms the state-of-the-art methods on non-differential queries while matching accuracy in approximating differentiable ones.
arXiv Detail & Related papers (2023-06-05T21:19:37Z) - From Robustness to Privacy and Back [15.727276506140878]
We study the relationship between two desiderata of algorithms in statistical inference and machine learning: differential privacy and robustness to adversarial data corruptions.
Our work gives the first black-box transformation that converts any adversarially robust algorithm into one that satisfies pure differential privacy.
arXiv Detail & Related papers (2023-02-03T16:57:57Z) - Private Domain Adaptation from a Public Source [48.83724068578305]
We design differentially private discrepancy-based algorithms for adaptation from a source domain with public labeled data to a target domain with unlabeled private data.
Our solutions are based on private variants of Frank-Wolfe and Mirror-Descent algorithms.
arXiv Detail & Related papers (2022-08-12T06:52:55Z) - Optimal Algorithms for Mean Estimation under Local Differential Privacy [55.32262879188817]
We show that PrivUnit achieves the optimal variance among a large family of locally private randomizers.
We also develop a new variant of PrivUnit based on the Gaussian distribution which is more amenable to mathematical analysis and enjoys the same optimality guarantees.
arXiv Detail & Related papers (2022-05-05T06:43:46Z) - Output Perturbation for Differentially Private Convex Optimization with
Improved Population Loss Bounds, Runtimes and Applications to Private
Adversarial Training [12.386462516398469]
Finding efficient, easily implementable differentially private (DP) algorithms that offer strong excess risk bounds is an important problem in modern machine learning.
We provide the tightest known $(epsilon, 0)$-DP population loss bounds and fastest runtimes under the presence of smoothness and strong convexity.
We apply our theory to two learning frameworks: tilted ERM and adversarial learning frameworks.
arXiv Detail & Related papers (2021-02-09T08:47:06Z) - Privacy Preserving Recalibration under Domain Shift [119.21243107946555]
We introduce a framework that abstracts out the properties of recalibration problems under differential privacy constraints.
We also design a novel recalibration algorithm, accuracy temperature scaling, that outperforms prior work on private datasets.
arXiv Detail & Related papers (2020-08-21T18:43:37Z) - Private Optimization Without Constraint Violations [14.920650598301476]
We study the problem of differentially private optimization with linear constraints when the right-hand-side of the constraints depends on private data.
Previous research provided solutions that retained privacy but sometimes violated the constraints.
We present an algorithm that releases a nearly-optimal solution satisfying the constraints with probability 1.
arXiv Detail & Related papers (2020-07-02T15:08:52Z) - Private Stochastic Non-Convex Optimization: Adaptive Algorithms and
Tighter Generalization Bounds [72.63031036770425]
We propose differentially private (DP) algorithms for bound non-dimensional optimization.
We demonstrate two popular deep learning methods on the empirical advantages over standard gradient methods.
arXiv Detail & Related papers (2020-06-24T06:01:24Z) - Private Stochastic Convex Optimization: Optimal Rates in Linear Time [74.47681868973598]
We study the problem of minimizing the population loss given i.i.d. samples from a distribution over convex loss functions.
A recent work of Bassily et al. has established the optimal bound on the excess population loss achievable given $n$ samples.
We describe two new techniques for deriving convex optimization algorithms both achieving the optimal bound on excess loss and using $O(minn, n2/d)$ gradient computations.
arXiv Detail & Related papers (2020-05-10T19:52:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.