Sequence-Level Leakage Risk of Training Data in Large Language Models

• URL: http://arxiv.org/abs/2412.11302v3
• Date: Wed, 16 Apr 2025 04:30:56 GMT
• Title: Sequence-Level Leakage Risk of Training Data in Large Language Models
• Authors: Trishita Tiwari, G. Edward Suh,
• Abstract summary: This work quantifies the risk of training data leakage from Large Language Models using sequence-level probabilities.<n>We re-analyze the effects of decoding schemes, model sizes, prefix lengths, partial sequence leakages, and token positions.
• Score: 7.600279942640982
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work quantifies the risk of training data leakage from LLMs (Large Language Models) using sequence-level probabilities. Computing extraction probabilities for individual sequences provides finer-grained information than has been studied in prior benchmarking work. We re-analyze the effects of decoding schemes, model sizes, prefix lengths, partial sequence leakages, and token positions to uncover new insights that were not possible in previous works due to their choice of metrics. We perform this study on two pre-trained models, Llama and OPT, trained on the Common Crawl and The Pile respectively. We discover that 1) Extraction Rate, the predominant metric used in prior quantification work, underestimates the threat of leakage of training data in randomized LLMs by as much as 2.14X. 2) Although on average, larger models and longer prefixes can extract more data, this is not true for a substantial portion of individual sequences. 30.4-41.5% of our sequences are easier to extract with either shorter prefixes or smaller models. 3) Contrary to previous beliefs, partial leakage in commonly used decoding schemes like top-k and top-p is not easier than leaking verbatim training data. The aim of this work is to encourage the adoption of this metric for future work on quantification of training data extraction.

Related papers

• How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning [2.3759432635713895]
  We attack both pre-trained and fine-tuned code language models to investigate the extent of data extractability. Fine-tuning requires fewer resources and is increasingly used by both small and large entities for its effectiveness on specialized data. Data carriers and licensing information are the most likely data to be memorized from pre-trained and fine-tuned models, while the latter is the most likely to be forgotten after fine-tuning.
  arXiv  Detail & Related papers  (2025-01-29T09:17:30Z)
• Training on the Benchmark Is Not All You Need [52.01920740114261]
  We propose a simple and effective data leakage detection method based on the contents of multiple-choice options. Our method is able to work under black-box conditions without access to model training data or weights. We evaluate the degree of data leakage of 31 mainstream open-source LLMs on four benchmark datasets.
  arXiv  Detail & Related papers  (2024-09-03T11:09:44Z)
• Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokens [1.2549198550400134]
  Large language models (LLMs) are extensively used, but there are concerns regarding privacy, security, and copyright due to their opaque training data. Current solutions to this problem leverage techniques explored in machine learning privacy such as Membership Inference Attacks (MIAs) We propose an adaptive pre-training data detection method which alleviates this reliance and effectively amplify the identification.
  arXiv  Detail & Related papers  (2024-07-30T23:43:59Z)
• Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data? [112.0422370149713]
  We tackle a task which we call data mixture inference, which aims to uncover the distributional make-up of training data.<n>We introduce a novel attack based on a previously overlooked source of information: byte-pair encoding (BPE) tokenizers.<n>We show that our attack recovers mixture ratios with high precision for tokenizers trained on known mixtures of natural languages, programming languages, and data sources.
  arXiv  Detail & Related papers  (2024-07-23T16:13:22Z)
• Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs [61.04246774006429]
  We introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent. We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements. Our findings show that instruction-tuned models can expose pre-training data as much as their base-models, if not more so, and using instructions proposed by other LLMs can open a new avenue of automated attacks.
  arXiv  Detail & Related papers  (2024-03-05T19:32:01Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [93.90047628101155]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To address this, some methods propose replaying data from previous tasks during new task learning. However, it is not expected in practice due to memory constraints and data privacy issues.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? [92.90857135952231]
  Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities. We study ICL in one of its simplest setups: pretraining a linearly parameterized single-layer linear attention model for linear regression.
  arXiv  Detail & Related papers  (2023-10-12T15:01:43Z)
• D4: Improving LLM Pretraining via Document De-Duplication and Diversification [38.84592304799403]
  We show that careful data selection via pre-trained model embeddings can speed up training. We also show that repeating data intelligently consistently outperforms baseline training.
  arXiv  Detail & Related papers  (2023-08-23T17:58:14Z)
• Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation [56.57532238195446]
  We propose a method named Ethicist for targeted training data extraction. To elicit memorization, we tune soft prompt embeddings while keeping the model fixed. We show that Ethicist significantly improves the extraction performance on a recently proposed public benchmark.
  arXiv  Detail & Related papers  (2023-07-10T08:03:41Z)
• Lifelong Language Pretraining with Distribution-Specialized Experts [39.86463645187337]
  Lifelong learning aims to enable information systems to learn from a continuous data stream across time. We propose Lifelong-MoE, an MoE architecture that dynamically adds model capacity via adding experts with regularized pretraining. Compared to existing lifelong learning approaches, Lifelong-MoE achieves better few-shot performance on 19 downstream NLP tasks.
  arXiv  Detail & Related papers  (2023-05-20T21:15:19Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.