Utility Assessment of Synthetic Data Generation Methods

• URL: http://arxiv.org/abs/2211.14428v1
• Date: Wed, 23 Nov 2022 11:09:52 GMT
• Title: Utility Assessment of Synthetic Data Generation Methods
• Authors: Md Sakib Nizam Khan, Niklas Reje, Sonja Buchegger
• Abstract summary: We investigate whether different methods of generating fully synthetic data vary in their utility a priori. We find some methods to perform better than others across the board. We do get promising findings for classification tasks when using synthetic data for training machine learning models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Big data analysis poses the dual problem of privacy preservation and utility, i.e., how accurate data analyses remain after transforming original data in order to protect the privacy of the individuals that the data is about - and whether they are accurate enough to be meaningful. In this paper, we thus investigate across several datasets whether different methods of generating fully synthetic data vary in their utility a priori (when the specific analyses to be performed on the data are not known yet), how closely their results conform to analyses on original data a posteriori, and whether these two effects are correlated. We find some methods (decision-tree based) to perform better than others across the board, sizeable effects of some choices of imputation parameters (notably the number of released datasets), no correlation between broad utility metrics and analysis accuracy, and varying correlations for narrow metrics. We did get promising findings for classification tasks when using synthetic data for training machine learning models, which we consider worth exploring further also in terms of mitigating privacy attacks against ML models such as membership inference and model inversion.

Related papers

• Inference With Combining Rules From Multiple Differentially Private Synthetic Datasets [0.0]
  We study the applicability of procedures based on combining rules to the analysis of DIPS datasets. Our empirical experiments show that the proposed combining rules may offer accurate inference in certain contexts, but not in all cases.
  arXiv  Detail & Related papers  (2024-05-08T02:33:35Z)
• Approximate, Adapt, Anonymize (3A): a Framework for Privacy Preserving Training Data Release for Machine Learning [3.29354893777827]
  We introduce a data release framework, 3A (Approximate, Adapt, Anonymize), to maximize data utility for machine learning. We present experimental evidence showing minimal discrepancy between performance metrics of models trained on real versus privatized datasets.
  arXiv  Detail & Related papers  (2023-07-04T18:37:11Z)
• Exploring the Effectiveness of Dataset Synthesis: An application of Apple Detection in Orchards [68.95806641664713]
  We explore the usability of Stable Diffusion 2.1-base for generating synthetic datasets of apple trees for object detection. We train a YOLOv5m object detection model to predict apples in a real-world apple detection dataset. Results demonstrate that the model trained on generated data is slightly underperforming compared to a baseline model trained on real-world images.
  arXiv  Detail & Related papers  (2023-06-20T09:46:01Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)
• Synthetic data generation for a longitudinal cohort study -- Evaluation, method extension and reproduction of published data analysis results [0.32593385688760446]
  In the health sector, access to individual-level data is often challenging due to privacy concerns. A promising alternative is the generation of fully synthetic data. In this study, we use a state-of-the-art synthetic data generation method.
  arXiv  Detail & Related papers  (2023-05-12T13:13:55Z)
• Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics [58.50754318846996]
  In this paper, we show that the performances of metrics are sensitive to data. The ranking of metrics varies when the evaluation is conducted on different datasets.
  arXiv  Detail & Related papers  (2022-03-29T18:58:28Z)
• Equivariance Allows Handling Multiple Nuisance Variables When Analyzing Pooled Neuroimaging Datasets [53.34152466646884]
  In this paper, we show how bringing recent results on equivariant representation learning instantiated on structured spaces together with simple use of classical results on causal inference provides an effective practical solution. We demonstrate how our model allows dealing with more than one nuisance variable under some assumptions and can enable analysis of pooled scientific datasets in scenarios that would otherwise entail removing a large portion of the samples.
  arXiv  Detail & Related papers  (2022-03-29T04:54:06Z)
• Data-SUITE: Data-centric identification of in-distribution incongruous examples [81.21462458089142]
  Data-SUITE is a data-centric framework to identify incongruous regions of in-distribution (ID) data. We empirically validate Data-SUITE's performance and coverage guarantees.
  arXiv  Detail & Related papers  (2022-02-17T18:58:31Z)
• Selecting the suitable resampling strategy for imbalanced data classification regarding dataset properties [62.997667081978825]
  In many application domains such as medicine, information retrieval, cybersecurity, social media, etc., datasets used for inducing classification models often have an unequal distribution of the instances of each class. This situation, known as imbalanced data classification, causes low predictive performance for the minority class examples. Oversampling and undersampling techniques are well-known strategies to deal with this problem by balancing the number of examples of each class.
  arXiv  Detail & Related papers  (2021-12-15T18:56:39Z)
• An Analysis of the Deployment of Models Trained on Private Tabular Synthetic Data: Unexpected Surprises [4.129847064263057]
  Diferentially private (DP) synthetic datasets are a powerful approach for training machine learning models. We study the effects of differentially private synthetic data generation on classification.
  arXiv  Detail & Related papers  (2021-06-15T21:00:57Z)
• Measuring Utility and Privacy of Synthetic Genomic Data [3.635321290763711]
  We provide the first evaluation of the utility and the privacy protection of five state-of-the-art models for generating synthetic genomic data. Overall, there is no single approach for generating synthetic genomic data that performs well across the board.
  arXiv  Detail & Related papers  (2021-02-05T17:41:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.