Synth-MIA: A Testbed for Auditing Privacy Leakage in Tabular Data Synthesis

• URL: http://arxiv.org/abs/2509.18014v1
• Date: Mon, 22 Sep 2025 16:53:38 GMT
• Title: Synth-MIA: A Testbed for Auditing Privacy Leakage in Tabular Data Synthesis
• Authors: Joshua Ward, Xiaofeng Lin, Chi-Hua Wang, Guang Cheng,
• Abstract summary: Tabular Generative Models are often argued to preserve privacy by creating synthetic datasets that resemble training data.<n>Membership Inference Attacks (MIAs) have recently emerged as a method for evaluating privacy leakage in synthetic data.<n>We propose a unified, model-agnostic threat framework that deploys a collection of attacks to estimate the maximum empirical privacy leakage in synthetic datasets.
• Score: 8.4361320391543
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Tabular Generative Models are often argued to preserve privacy by creating synthetic datasets that resemble training data. However, auditing their empirical privacy remains challenging, as commonly used similarity metrics fail to effectively characterize privacy risk. Membership Inference Attacks (MIAs) have recently emerged as a method for evaluating privacy leakage in synthetic data, but their practical effectiveness is limited. Numerous attacks exist across different threat models, each with distinct implementations targeting various sources of privacy leakage, making them difficult to apply consistently. Moreover, no single attack consistently outperforms the others, leading to a routine underestimation of privacy risk. To address these issues, we propose a unified, model-agnostic threat framework that deploys a collection of attacks to estimate the maximum empirical privacy leakage in synthetic datasets. We introduce Synth-MIA, an open-source Python library that streamlines this auditing process through a novel testbed that integrates seamlessly into existing synthetic data evaluation pipelines through a Scikit-Learn-like API. Our software implements 13 attack methods through a Scikit-Learn-like API, designed to enable fast systematic estimation of privacy leakage for practitioners as well as facilitate the development of new attacks and experiments for researchers. We demonstrate our framework's utility in the largest tabular synthesis privacy benchmark to date, revealing that higher synthetic data quality corresponds to greater privacy leakage, that similarity-based privacy metrics show weak correlation with MIA results, and that the differentially private generator PATEGAN can fail to preserve privacy under such attacks. This underscores the necessity of MIA-based auditing when designing and deploying Tabular Generative Models.

Related papers

• SynQP: A Framework and Metrics for Evaluating the Quality and Privacy Risk of Synthetic Data [4.73374389278596]
  We introduce SynQP, an open framework for privacy benchmarking in synthetic data generation.<n>We also highlight the need for privacy metrics that fairly account for the probabilistic nature of machine learning models.<n>Our work provides a critical tool for improving the transparency and reliability of privacy evaluations.
  arXiv  Detail & Related papers  (2026-01-17T17:51:14Z)
• How to DP-fy Your Data: A Practical Guide to Generating Synthetic Data With Differential Privacy [52.00934156883483]
  Differential Privacy (DP) is a framework for reasoning about and limiting information leakage.<n>Differentially Private Synthetic data refers to synthetic data that preserves the overall trends of source data.
  arXiv  Detail & Related papers  (2025-12-02T21:14:39Z)
• PRIVET: Privacy Metric Based on Extreme Value Theory [8.447463478355845]
  Deep generative models are often trained on sensitive data, such as genetic sequences, health data, or more broadly, any copyrighted, licensed or protected content.<n>This raises critical concerns around privacy-preserving synthetic data, and more specifically around privacy leakage.<n>We propose PRIVET, a generic sample-based, modality-agnostic algorithm that assigns an individual privacy leak score to each synthetic sample.
  arXiv  Detail & Related papers  (2025-10-28T09:42:03Z)
• On the MIA Vulnerability Gap Between Private GANs and Diffusion Models [51.53790101362898]
  Generative Adversarial Networks (GANs) and diffusion models have emerged as leading approaches for high-quality image synthesis.<n>We present the first unified theoretical and empirical analysis of the privacy risks faced by differentially private generative models.
  arXiv  Detail & Related papers  (2025-09-03T14:18:22Z)
• Privacy Auditing Synthetic Data Release through Local Likelihood Attacks [7.780592134085148]
  Gene Likelihood Ratio Attack (Gen-LRA)<n>Gen-LRA formulates its attack by evaluating the influence a test observation has in a surrogate model's estimation of a local likelihood ratio over the synthetic data.<n>Results underscore Gen-LRA's effectiveness as a privacy auditing tool for the release of synthetic data.
  arXiv  Detail & Related papers  (2025-08-28T18:27:40Z)
• PriRoAgg: Achieving Robust Model Aggregation with Minimum Privacy Leakage for Federated Learning [49.916365792036636]
  Federated learning (FL) has recently gained significant momentum due to its potential to leverage large-scale distributed user data.<n>The transmitted model updates can potentially leak sensitive user information, and the lack of central control of the local training process leaves the global model susceptible to malicious manipulations on model updates.<n>We develop a general framework PriRoAgg, utilizing Lagrange coded computing and distributed zero-knowledge proof, to execute a wide range of robust aggregation algorithms while satisfying aggregated privacy.
  arXiv  Detail & Related papers  (2024-07-12T03:18:08Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• FewFedPIT: Towards Privacy-preserving and Few-shot Federated Instruction Tuning [54.26614091429253]
  Federated instruction tuning (FedIT) is a promising solution, by consolidating collaborative training across multiple data owners. FedIT encounters limitations such as scarcity of instructional data and risk of exposure to training data extraction attacks. We propose FewFedPIT, designed to simultaneously enhance privacy protection and model performance of federated few-shot learning.
  arXiv  Detail & Related papers  (2024-03-10T08:41:22Z)
• The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against "Truly Anonymous" Synthetic Datasets [12.730435519914415]
  We examine the privacy metrics used in real-world synthetic data deployments and demonstrate their unreliability in several ways.<n>We introduce ReconSyn, a reconstruction attack that generates multiple synthetic datasets that are considered private by the metrics but actually leak unique information to individual records.<n>We show that ReconSyn recovers 78-100% of the outliers in the train data with only black-box access to a single fitted generative model and the privacy metrics.
  arXiv  Detail & Related papers  (2023-12-08T15:42:28Z)
• Achilles' Heels: Vulnerable Record Identification in Synthetic Data Publishing [9.061271587514215]
  We propose a principled vulnerable record identification technique for synthetic data publishing. We show it to strongly outperform previous ad-hoc methods across datasets and generators. We show it to accurately identify vulnerable records when synthetic data generators are made differentially private.
  arXiv  Detail & Related papers  (2023-06-17T09:42:46Z)
• Auditing and Generating Synthetic Data with Controllable Trust Trade-offs [54.262044436203965]
  We introduce a holistic auditing framework that comprehensively evaluates synthetic datasets and AI models. It focuses on preventing bias and discrimination, ensures fidelity to the source data, assesses utility, robustness, and privacy preservation. We demonstrate the framework's effectiveness by auditing various generative models across diverse use cases.
  arXiv  Detail & Related papers  (2023-04-21T09:03:18Z)
• Membership Inference Attacks against Synthetic Data through Overfitting Detection [84.02632160692995]
  We argue for a realistic MIA setting that assumes the attacker has some knowledge of the underlying data distribution. We propose DOMIAS, a density-based MIA model that aims to infer membership by targeting local overfitting of the generative model.
  arXiv  Detail & Related papers  (2023-02-24T11:27:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.