Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?

• URL: http://arxiv.org/abs/2403.13612v2
• Date: Fri, 23 Aug 2024 09:53:37 GMT
• Title: Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?
• Authors: Ileana Montoya Perez, Parisa Movahedi, Valtteri Nieminen, Antti Airola, Tapio Pahikkala,
• Abstract summary: The evaluation is conducted in terms of the tests' Type I and Type II errors. A large portion of the evaluation results expressed dramatically inflated Type I errors, especially at privacy budget levels of $epsilonleq 1$.
• Score: 1.9573380763700712
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Background: Synthetic data has been proposed as a solution for sharing anonymized versions of sensitive biomedical datasets. Ideally, synthetic data should preserve the structure and statistical properties of the original data, while protecting the privacy of the individual subjects. Differential privacy (DP) is currently considered the gold standard approach for balancing this trade-off. Objectives: To investigate the reliability of group differences identified by independent sample tests on DP-synthetic data. The evaluation is conducted in terms of the tests' Type I and Type II errors. The former quantifies the tests' validity i.e. whether the probability of false discoveries is indeed below the significance level, and the latter indicates the tests' power in making real discoveries. Methods: We evaluate the Mann-Whitney U test, Student's t-test, chi-squared test and median test on DP-synthetic data. The private synthetic datasets are generated from real-world data, including a prostate cancer dataset (n=500) and a cardiovascular dataset (n=70 000), as well as on bivariate and multivariate simulated data. Five different DP-synthetic data generation methods are evaluated, including two basic DP histogram release methods and MWEM, Private-PGM, and DP GAN algorithms. Conclusion: A large portion of the evaluation results expressed dramatically inflated Type I errors, especially at privacy budget levels of $\epsilon\leq 1$. This result calls for caution when releasing and analyzing DP-synthetic data: low p-values may be obtained in statistical tests simply as a byproduct of the noise added to protect privacy. A DP smoothed histogram-based synthetic data generation method was shown to produce valid Type I error for all privacy levels tested but required a large original dataset size and a modest privacy budget ($\epsilon\geq 5$) in order to have reasonable Type II error.

Related papers

• Federated Causal Discovery from Heterogeneous Data [70.31070224690399]
  We propose a novel FCD method attempting to accommodate arbitrary causal models and heterogeneous data. These approaches involve constructing summary statistics as a proxy of the raw data to protect data privacy. We conduct extensive experiments on synthetic and real datasets to show the efficacy of our method.
  arXiv  Detail & Related papers  (2024-02-20T18:53:53Z)
• Benchmarking Private Population Data Release Mechanisms: Synthetic Data vs. TopDown [50.40020716418472]
  This study conducts a comparison between the TopDown algorithm and private synthetic data generation to determine how accuracy is affected by query complexity. Our results show that for in-distribution queries, the TopDown algorithm achieves significantly better privacy-fidelity tradeoffs than any of the synthetic data methods we evaluated.
  arXiv  Detail & Related papers  (2024-01-31T17:38:34Z)
• DP-PQD: Privately Detecting Per-Query Gaps In Synthetic Data Generated By Black-Box Mechanisms [17.562365686511818]
  We present a novel framework named DP-PQD (differentially-private per-query decider) to detect if the query answers on the private and synthetic datasets are within a user-specified threshold of each other. We give a suite of private algorithms for per-query deciders for count, sum, and median queries, analyze their properties, and evaluate them experimentally.
  arXiv  Detail & Related papers  (2023-09-15T17:38:59Z)
• Mean Estimation with User-level Privacy under Data Heterogeneity [54.07947274508013]
  Different users may possess vastly different numbers of data points. It cannot be assumed that all users sample from the same underlying distribution. We propose a simple model of heterogeneous user data that allows user data to differ in both distribution and quantity of data.
  arXiv  Detail & Related papers  (2023-07-28T23:02:39Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)
• Membership Inference Attacks against Synthetic Data through Overfitting Detection [84.02632160692995]
  We argue for a realistic MIA setting that assumes the attacker has some knowledge of the underlying data distribution. We propose DOMIAS, a density-based MIA model that aims to infer membership by targeting local overfitting of the generative model.
  arXiv  Detail & Related papers  (2023-02-24T11:27:39Z)
• Utility Assessment of Synthetic Data Generation Methods [0.0]
  We investigate whether different methods of generating fully synthetic data vary in their utility a priori. We find some methods to perform better than others across the board. We do get promising findings for classification tasks when using synthetic data for training machine learning models.
  arXiv  Detail & Related papers  (2022-11-23T11:09:52Z)
• Noise-Aware Statistical Inference with Differentially Private Synthetic Data [0.0]
  We show that simply analysing DP synthetic data as if it were real does not produce valid inferences of population-level quantities. We tackle this problem by combining synthetic data analysis techniques from the field of multiple imputation, and synthetic data generation. We develop a novel noise-aware synthetic data generation algorithm NAPSU-MQ using the principle of maximum entropy.
  arXiv  Detail & Related papers  (2022-05-28T16:59:46Z)
• DTGAN: Differential Private Training for Tabular GANs [6.174448419090292]
  We propose DTGAN, a novel conditional Wasserstein GAN that comes in two variants DTGAN_G and DTGAN_D. We rigorously evaluate the theoretical privacy guarantees offered by DP empirically against membership and attribute inference attacks. Our results on 3 datasets show that the DP-SGD framework is superior to PATE and that a DP discriminator is more optimal for training convergence.
  arXiv  Detail & Related papers  (2021-07-06T10:28:05Z)
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data [62.29031007761901]
  This paper proposes a novel contrastive regularized clinical classification model. We introduce two unique positive sampling strategies specifically tailored for EHR data. Our framework yields highly competitive experimental results in predicting the mortality risk on real-world COVID-19 EHR data.
  arXiv  Detail & Related papers  (2021-04-07T06:02:04Z)
• Generation of Differentially Private Heterogeneous Electronic Health Records [9.926231893220061]
  We explore using Generative Adversarial Networks to generate synthetic, heterogeneous EHRs. We will explore applying differential privacy (DP) preserving optimization in order to produce DP synthetic EHR data sets.
  arXiv  Detail & Related papers  (2020-06-05T13:21:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.