Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models

• URL: http://arxiv.org/abs/2608.00144v1
• Date: Fri, 31 Jul 2026 15:44:43 GMT
• Title: Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models
• Abstract summary: We study black-box, sampling-based training-data leakage through a probabilistic lens.<n>We extend the blind-baseline critique into the sampling regime.<n>We release leakit, a black-box extraction-audit tool.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Membership inference (MIA) on language models is usually summarised by an aggregate ROC-AUC, but such evaluations are confounded: model-free blind baselines separate members from non-members from surface text alone. We study black-box, sampling-based training-data leakage through a probabilistic lens, treating N samples from p(.|x) as an estimate of the output distribution and casting leakage signals as functionals of it. We extend the blind-baseline critique into the sampling regime: on WikiMIA a blind bag-of-words classifier reaches AUC 0.97 (TPR 0.90 at 5% FPR) and sampling adds nothing, while on an IID Pile split (MIMIR) neither self-concentration nor gold-continuation recovery significantly beats a blind baseline (incremental AUC 95% CI includes zero). Aggregate metrics hide the real harm. The same sampling verbatim-extracts training data for a tail of documents no blind attack can reach. On Pythia-6.9B, 83 of 500 Pile documents bearing a real identifier (16.6%; 21.3% of those bearing an email address) have that exact identifier reproduced AND not reproduced under a mismatched-prefix control, so each leak is attributable to that document, not to a globally common string. This per-document disclosure is invisible to aggregate AUC and grows with capacity (5.6% to 16.6% from 410M to 6.9B). The risk is uneven: identifier leakage is ~3x stronger in code than prose, though prose stays clearly positive and also grows with capacity (4.0% to 12.1%), while recovery of arbitrary held-out continuations is confined to code (+0.44 member gap on GitHub vs at most +0.014 on prose). Temperature and nucleus sampling matter little, a 16-token prefix suffices, and we detect no reduction from corpus deduplication. Privacy audits should report per-document extraction, decomposed by domain, not a single AUC. We release leakit, a black-box extraction-audit tool.

Related papers

• The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context [45.88028371034407]
  We report a controlled case study in code repair, not a new phenomenon.<n>We toggle one flag (one-chunk-perfile deduplication) on an otherwise identical stack.<n>We map where it holds: it reverses on a lexical BM25 (3.2pp, significant cross-paradigm interaction)<n>We show that at a tight fixed budget, do not hard-deduplicate by file, and A/B packing policies against the task, not the metric flag was tuned.
  arXiv  Detail & Related papers  (2026-08-14T19:22:50Z)
• From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts [4.093009863847175]
  Security research artifacts are increasingly produced by agent-driven vulnerabilities.<n>The gap between emphpublicly available, emphrunnable, emphsignal-producing, and emphsemantically confirmed artifacts is poorly measured.<n>We conduct a pre-registered consensus audit of this literature.
  arXiv  Detail & Related papers  (2026-08-10T13:05:02Z)
• Beyond Accuracy: Auditing Spatial Provenance in Visual Token Pruning for OCR-Critical MLLM Inference [22.335515110971475]
  Visual-token pruning is judged by answer quality at a fixed retention budget.<n>We turn this blind spot into an evidence-risk audit that couples answer behavior with geometric token-origin provenance.<n>We show that favorable target-verification points do not imply task-general compression.
  arXiv  Detail & Related papers  (2026-07-29T13:45:31Z)
• Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion [46.262619407930266]
  We train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings.<n>The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline.<n>We then investigate whether LLM-assisted labeling can extend the gold dataset.
  arXiv  Detail & Related papers  (2026-07-28T10:59:04Z)
• Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction [2.1997815944423516]
  ExtractConf is a cross-domain, field-agnostic confidence engine.<n>It grounds confidence estimation in two structurally different readings of the same document.<n>It achieves 0.928 ROC AUC and reduces selective prediction risk by 70% over logprob-mean.
  arXiv  Detail & Related papers  (2026-06-23T10:58:08Z)
• Now You (Still) See Me: Detecting Evasive Steganographic Payloads in LLMs [52.149036302760386]
  Large language models can be fine-tuned to encode prompt-borne secrets into fluent, seemingly benign outputs.<n>Recent work proposes detection using linear probes that recover the secret from internal activations.<n>We show that this defense can be systematically evaded, but that detectability can be recovered through a targeted data-level intervention.
  arXiv  Detail & Related papers  (2026-06-08T12:27:11Z)
• SilentRetrieval: Hijacking Retrieval-Augmented Generation via Semantically-Preserving Adversarial Data Poisoning [1.0998907972211756]
  We present SilentRetrieval, a two-stage data poisoning attack that hijacks RAG systems through adversarially crafted documents.<n>SilentRetrieval achieves 84.6%/81.3% HR@10 and 57.5%/54.8% ASR-LLM on Natural Questions and MS MARCO, while maintaining near-benign perplexity.
  arXiv  Detail & Related papers  (2026-05-27T07:30:30Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• What Are We Actually Decoding? Source Attribution for Non-Invasive Brain-to-Language Retrieval [42.66754319854329]
  We recast stimulus-locked MEG-to-audio retrieval as an auditing framework.<n>We use structural shortcuts, window-level stimulus-locked evidence, and cross-window contextual aggregation.<n>These results suggest that brain-to-language performance should be source-attributed, not merely reported.
  arXiv  Detail & Related papers  (2026-05-23T11:23:39Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions [50.1404916337174]
  We present the first large-scale, statistically powered audit of native probabilistic sampling in large language models (LLMs)<n>We show that batch generation achieves only modest statistical validity, with a 13% median pass rate, while independent requests collapse almost entirely.<n>We conclude that current LLMs lack a functional internal sampler, necessitating the use of external tools for applications requiring statistical guarantees.
  arXiv  Detail & Related papers  (2026-01-08T22:33:12Z)
• Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models [81.62767292169225]
  We investigate knowledge forgetting in large language models with a focus on its generalisation.<n>We propose PerMU, a novel probability perturbation-based unlearning paradigm.<n>Experiments are conducted on a diverse range of datasets, including TOFU, Harry Potter, ZsRE, WMDP, and MUSE.
  arXiv  Detail & Related papers  (2025-02-27T11:03:33Z)
• Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method [108.56493934296687]
  We introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection.<n>We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.
  arXiv  Detail & Related papers  (2024-09-23T07:55:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.