Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?

• URL: http://arxiv.org/abs/2403.13612v1
• Date: Wed, 20 Mar 2024 14:03:57 GMT
• Title: Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?
• Authors: Ileana Montoya Perez, Parisa Movahedi, Valtteri Nieminen, Antti Airola, Tapio Pahikkala,
• Abstract summary: Differential privacy (DP) is currently considered the gold standard approach for balancing this trade-off. This study is to evaluate the Mann-Whitney U test on DP-synthetic biomedical data in terms of Type I and Type II errors.
• Score: 1.9573380763700712
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Background: Synthetic data has been proposed as a solution for sharing anonymized versions of sensitive biomedical datasets. Ideally, synthetic data should preserve the structure and statistical properties of the original data, while protecting the privacy of the individual subjects. Differential privacy (DP) is currently considered the gold standard approach for balancing this trade-off. Objectives: The aim of this study is to evaluate the Mann-Whitney U test on DP-synthetic biomedical data in terms of Type I and Type II errors, in order to establish whether statistical hypothesis testing performed on privacy preserving synthetic data is likely to lead to loss of test's validity or decreased power. Methods: We evaluate the Mann-Whitney U test on DP-synthetic data generated from real-world data, including a prostate cancer dataset (n=500) and a cardiovascular dataset (n=70 000), as well as on data drawn from two Gaussian distributions. Five different DP-synthetic data generation methods are evaluated, including two basic DP histogram release methods and MWEM, Private-PGM, and DP GAN algorithms. Conclusion: Most of the tested DP-synthetic data generation methods showed inflated Type I error, especially at privacy budget levels of $\epsilon\leq 1$. This result calls for caution when releasing and analyzing DP-synthetic data: low p-values may be obtained in statistical tests simply as a byproduct of the noise added to protect privacy. A DP smoothed histogram-based synthetic data generation method was shown to produce valid Type I error for all privacy levels tested but required a large original dataset size and a modest privacy budget ($\epsilon\geq 5$) in order to have reasonable Type II error levels.

Related papers

• Improving Grammatical Error Correction via Contextual Data Augmentation [49.746484518527716]
  We propose a synthetic data construction method based on contextual augmentation. Specifically, we combine rule-based substitution with model-based generation. We also propose a relabeling-based data cleaning method to mitigate the effects of noisy labels in synthetic data.
  arXiv  Detail & Related papers  (2024-06-25T10:49:56Z)
• Benchmarking Private Population Data Release Mechanisms: Synthetic Data vs. TopDown [50.40020716418472]
  This study conducts a comparison between the TopDown algorithm and private synthetic data generation to determine how accuracy is affected by query complexity. Our results show that for in-distribution queries, the TopDown algorithm achieves significantly better privacy-fidelity tradeoffs than any of the synthetic data methods we evaluated.
  arXiv  Detail & Related papers  (2024-01-31T17:38:34Z)
• Differentially Private Data Generation with Missing Data [25.242190235853595]
  We formalize the problems of differential privacy (DP) synthetic data with missing values. We propose three effective adaptive strategies that significantly improve the utility of the synthetic data. Overall, this study contributes to a better understanding of the challenges and opportunities for using private synthetic data generation algorithms.
  arXiv  Detail & Related papers  (2023-10-17T19:41:54Z)
• DP-PQD: Privately Detecting Per-Query Gaps In Synthetic Data Generated By Black-Box Mechanisms [17.562365686511818]
  We present a novel framework named DP-PQD (differentially-private per-query decider) to detect if the query answers on the private and synthetic datasets are within a user-specified threshold of each other. We give a suite of private algorithms for per-query deciders for count, sum, and median queries, analyze their properties, and evaluate them experimentally.
  arXiv  Detail & Related papers  (2023-09-15T17:38:59Z)
• Mean Estimation with User-level Privacy under Data Heterogeneity [54.07947274508013]
  Different users may possess vastly different numbers of data points. It cannot be assumed that all users sample from the same underlying distribution. We propose a simple model of heterogeneous user data that allows user data to differ in both distribution and quantity of data.
  arXiv  Detail & Related papers  (2023-07-28T23:02:39Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)
• Membership Inference Attacks against Synthetic Data through Overfitting Detection [84.02632160692995]
  We argue for a realistic MIA setting that assumes the attacker has some knowledge of the underlying data distribution. We propose DOMIAS, a density-based MIA model that aims to infer membership by targeting local overfitting of the generative model.
  arXiv  Detail & Related papers  (2023-02-24T11:27:39Z)
• PreFair: Privately Generating Justifiably Fair Synthetic Data [17.037575948075215]
  PreFair is a system that allows for Differential Privacy (DP) fair synthetic data generation. We adapt the notion of justifiable fairness to fit the synthetic data generation scenario.
  arXiv  Detail & Related papers  (2022-12-20T15:01:54Z)
• Noise-Aware Statistical Inference with Differentially Private Synthetic Data [0.0]
  We show that simply analysing DP synthetic data as if it were real does not produce valid inferences of population-level quantities. We tackle this problem by combining synthetic data analysis techniques from the field of multiple imputation, and synthetic data generation. We develop a novel noise-aware synthetic data generation algorithm NAPSU-MQ using the principle of maximum entropy.
  arXiv  Detail & Related papers  (2022-05-28T16:59:46Z)
• DTGAN: Differential Private Training for Tabular GANs [6.174448419090292]
  We propose DTGAN, a novel conditional Wasserstein GAN that comes in two variants DTGAN_G and DTGAN_D. We rigorously evaluate the theoretical privacy guarantees offered by DP empirically against membership and attribute inference attacks. Our results on 3 datasets show that the DP-SGD framework is superior to PATE and that a DP discriminator is more optimal for training convergence.
  arXiv  Detail & Related papers  (2021-07-06T10:28:05Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.