Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection

• URL: http://arxiv.org/abs/2501.11786v1
• Date: Mon, 20 Jan 2025 23:19:15 GMT
• Title: Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection
• Authors: Ali Naseh, Niloofar Mireshghallah,
• Abstract summary: Using synthetic data in membership evaluations may lead to false conclusions about model memorization and data leakage.<n>This issue could affect other evaluations using model signals such as loss where synthetic or machine-generated translated data substitutes for real-world samples.
• Score: 1.03590082373586
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work shows membership inference attacks (MIAs) on large language models (LLMs) produce inconclusive results, partly due to difficulties in creating non-member datasets without temporal shifts. While researchers have turned to synthetic data as an alternative, we show this approach can be fundamentally misleading. Our experiments indicate that MIAs function as machine-generated text detectors, incorrectly identifying synthetic data as training samples regardless of the data source. This behavior persists across different model architectures and sizes, from open-source models to commercial ones such as GPT-3.5. Even synthetic text generated by different, potentially larger models is classified as training data by the target model. Our findings highlight a serious concern: using synthetic data in membership evaluations may lead to false conclusions about model memorization and data leakage. We caution that this issue could affect other evaluations using model signals such as loss where synthetic or machine-generated translated data substitutes for real-world samples.

Related papers

• AI-Generated Fall Data: Assessing LLMs and Diffusion Model for Wearable Fall Detection [3.5912245880418125]
  Training fall detection systems is challenging due to the scarcity of real-world fall data, particularly from elderly individuals.<n>This study evaluates text-to-motion and text-to-text models in simulating realistic fall scenarios.<n>We generate synthetic datasets and integrate them with four real-world baseline datasets to assess their impact on fall detection performance.
  arXiv  Detail & Related papers  (2025-05-07T02:30:33Z)
• The Comparability of Model Fusion to Measured Data in Confuser Rejection [0.24578723416255746]
  No dataset can account for every slight deviation we might see in live usage.<n>Simulators have been developed utilizing the shooting and bouncing ray method to allow for the generation of synthetic SAR data on 3D models.<n>We aim to use computational power as a substitution for this lack of quality measured data, by ensembling many models trained on synthetic data.
  arXiv  Detail & Related papers  (2025-05-01T19:51:30Z)
• Assessing Generative Models for Structured Data [0.0]
  This paper introduces rigorous methods for assessing synthetic data against real data by looking at inter-column dependencies within the data. We find that large language models (GPT-2), both when queried via few-shot prompting, and when fine-tuned, and GAN (CTGAN) models do not produce data with dependencies that mirror the original real data.
  arXiv  Detail & Related papers  (2025-03-26T18:19:05Z)
• The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text [23.412546862849396]
  We design membership inference attacks (MIAs) that target data used to fine-tune pre-trained Large Language Models (LLMs) We show that such data-based MIAs do significantly better than a random guess, meaning that synthetic data leaks information about the training data. To tackle this problem, we leverage the mechanics of auto-regressive models to design canaries with an in-distribution prefix and a high-perplexity suffix.
  arXiv  Detail & Related papers  (2025-02-19T15:30:30Z)
• How to Synthesize Text Data without Model Collapse? [37.219627817995054]
  Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance.<n>We propose token editing on human-produced data to obtain semi-synthetic data.
  arXiv  Detail & Related papers  (2024-12-19T09:43:39Z)
• Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World [19.266191284270793]
  generative machine learning models are pretrained on web-scale datasets containing data generated by earlier models. Some prior work warns of "model collapse" as the web is overwhelmed by synthetic data. We report experiments on three ways of using data (training-workflows) across three generative model task-settings.
  arXiv  Detail & Related papers  (2024-10-22T05:49:24Z)
• Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models [89.88010750772413]
  Synthetic data has been proposed as a solution to address the issue of high-quality data scarcity in the training of large language models (LLMs) Our work delves into these specific flaws associated with question-answer (Q-A) pairs, a prevalent type of synthetic data, and presents a method based on unlearning techniques to mitigate these flaws. Our work has yielded key insights into the effective use of synthetic data, aiming to promote more robust and efficient LLM training.
  arXiv  Detail & Related papers  (2024-06-18T08:38:59Z)
• Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification [11.6055501181235]
  We investigate the use of verification on synthesized data to prevent model collapse. We show that verifiers, even imperfect ones, can indeed be harnessed to prevent model collapse.
  arXiv  Detail & Related papers  (2024-06-11T17:46:16Z)
• Learning Defect Prediction from Unrealistic Data [57.53586547895278]
  Pretrained models of code have become popular choices for code understanding and generation tasks. Such models tend to be large and require commensurate volumes of training data. It has become popular to train models with far larger but less realistic datasets, such as functions with artificially injected bugs. Models trained on such data tend to only perform well on similar data, while underperforming on real world programs.
  arXiv  Detail & Related papers  (2023-11-02T01:51:43Z)
• Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models [69.76066070227452]
  *Data Synthesis* is a promising way to train a small model with very little labeled data. We propose *Synthesis Step by Step* (**S3**), a data synthesis framework that shrinks this distribution gap. Our approach improves the performance of a small model by reducing the gap between the synthetic dataset and the real data.
  arXiv  Detail & Related papers  (2023-10-20T17:14:25Z)
• From Zero to Hero: Detecting Leaked Data through Synthetic Data Injection and Model Querying [10.919336198760808]
  We introduce a novel methodology to detect leaked data that are used to train classification models. textscLDSS involves injecting a small volume of synthetic data--characterized by local shifts in class distribution--into the owner's dataset. This enables the effective identification of models trained on leaked data through model querying alone.
  arXiv  Detail & Related papers  (2023-10-06T10:36:28Z)
• Synthetic is all you need: removing the auxiliary data assumption for membership inference attacks against synthetic data [9.061271587514215]
  We show how this assumption can be removed, allowing for MIAs to be performed using only the synthetic data. Our results show that MIAs are still successful, across two real-world datasets and two synthetic data generators.
  arXiv  Detail & Related papers  (2023-07-04T13:16:03Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)
• Membership Inference Attacks against Synthetic Data through Overfitting Detection [84.02632160692995]
  We argue for a realistic MIA setting that assumes the attacker has some knowledge of the underlying data distribution. We propose DOMIAS, a density-based MIA model that aims to infer membership by targeting local overfitting of the generative model.
  arXiv  Detail & Related papers  (2023-02-24T11:27:39Z)
• Advancing Semi-Supervised Learning for Automatic Post-Editing: Data-Synthesis by Mask-Infilling with Erroneous Terms [5.366354612549173]
  We focus on data-synthesis methods to create high-quality synthetic data. We present a data-synthesis method by which the resulting synthetic data mimic the translation errors found in actual data. Experimental results show that using the synthetic data created by our approach results in significantly better APE performance than other synthetic data created by existing methods.
  arXiv  Detail & Related papers  (2022-04-08T07:48:57Z)
• Contrastive Model Inversion for Data-Free Knowledge Distillation [60.08025054715192]
  We propose Contrastive Model Inversion, where the data diversity is explicitly modeled as an optimizable objective. Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI achieves significantly superior performance when the generated data are used for knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-18T15:13:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.