Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs

• URL: http://arxiv.org/abs/2604.18697v1
• Date: Mon, 20 Apr 2026 18:00:22 GMT
• Title: Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs
• Abstract summary: We show that indistinguishability properties are neither sufficient nor necessary for preventing data extraction in LLM API.<n>We formalize a privacy-game separation between extraction and indistinguishability-based privacy.<n>We provide actionable mitigation guidelines across model training, API access, and decoding configurations.
• Score: 12.914333266854102
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Indistinguishability properties such as differential privacy bounds or low empirically measured membership inference are widely treated as proxies to show a model is sufficiently protected against broader memorization risks. However, we show that indistinguishability properties are neither sufficient nor necessary for preventing data extraction in LLM APIs. We formalize a privacy-game separation between extraction and indistinguishability-based privacy, showing that indistinguishability and inextractability are incomparable: upper-bounding distinguishability does not upper-bound extractability. To address this gap, we introduce $(l, b)$-inextractability as a definition that requires at least $2^b$ expected queries for any black-box adversary to induce the LLM API to emit a protected $l$-gram substring. We instantiate this via a worst-case extraction game and derive a rank-based extraction risk upper bound for targeted exact extraction, as well as extensions to cover untargeted and approximate extraction. The resulting estimator captures the extraction risk over multiple attack trials and prefix adaptations. We show that it can provide a tight and efficient estimation for standard greedy extraction and an upper bound on the probabilistic extraction risk given any decoding configuration. We empirically evaluate extractability across different models, clarifying its connection to distinguishability, demonstrating its advantage over existing extraction risk estimators, and providing actionable mitigation guidelines across model training, API access, and decoding configurations in LLM API deployment. Our code is publicly available at: https://github.com/Emory-AIMS/Inextractability.

Related papers

• Sound Probabilistic Safety Bounds for Large Language Models [50.060741829250645]
  We study a new application of the Clopper-Pearson confidence intervals to obtain probably approximately correct (PAC) bounds for this problem.<n>Our approach in particular enables the efficient computation of useful lower bounds, even in scenarios where the true harm probability is extremely small.
  arXiv  Detail & Related papers  (2026-07-22T15:31:28Z)
• From Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability [53.681678236115836]
  We study when can an SAE-based explanation be treated as a faithful view of an underlying frozen LM.<n>Our framework derives an upper bound on the base model's expected risk using four measurable quantities.<n>We show that the bound becomes non-vacuous on GPT-2 Small, Gemma-2B, and Llama-3-8B at practical sample sizes.
  arXiv  Detail & Related papers  (2026-06-16T18:28:23Z)
• An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic [16.151531550259993]
  Large language models (LLMs) are increasingly deployed through hosted APIs, making model extraction a practical threat to model ownership and service security.<n>We formulate model extraction monitoring as benign-calibrated traffic-window distribution testing.<n>We show that an embarrassingly simple detector is effective: embed incoming queries into a semantic space and test whether their aggregate distribution deviates from historical benign traffic.
  arXiv  Detail & Related papers  (2026-06-04T05:33:49Z)
• $k$NNProxy: Efficient Training-Free Proxy Alignment for Black-Box Zero-Shot LLM-Generated Text Detection [19.213077720525696]
  Existing LGT detectors can be classified into two broad classes: learning-based approaches and zero-shot methods.<n>The reliability of zero-shot methods relies on the assumption that an off-the-shelf proxy LLM is well aligned with the often unknown source.<n>We propose the $k$-nearest neighbor proxy ($k$NN Proxy), a training-free and query-efficient proxy alignment framework.
  arXiv  Detail & Related papers  (2026-04-02T13:11:06Z)
• Estimating near-verbatim extraction risk in language models with decoding-constrained beam search [66.14049361268074]
  We show that greedy-decoding methods miss how extraction risk varies across sequences.<n>We introduce decoding-constrained beam search, which yields deterministic lower bounds on near-verbatim extraction risk.
  arXiv  Detail & Related papers  (2026-03-26T01:15:16Z)
• Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model [99.16364381244445]
  Modern language models (LMs) tend to memorize portions of their training data and emit verbatim spans.<n>We propose Anchored Decoding, a plug-and-play inference-time method for suppressing verbatim copying.<n>We evaluate our methods across six model pairs on long-form evaluations of copyright risk and utility.
  arXiv  Detail & Related papers  (2026-02-06T19:00:14Z)
• VortexPIA: Indirect Prompt Injection Attack against LLMs for Efficient Extraction of User Privacy [22.037235521470468]
  Large language models (LLMs) have been widely deployed in Conversational AIs (CAIs)<n>Recent research shows that LLM-based CAIs can be manipulated to extract private information from human users, posing serious security threats.<n>We propose textscVortexPIA, a novel indirect prompt injection attack that induces privacy extraction under black-box settings.
  arXiv  Detail & Related papers  (2025-10-05T15:58:55Z)
• SafePassage: High-Fidelity Information Extraction with Black Box LLMs [6.5404806332891114]
  Black box large language models (LLMs) make information extraction (IE) easy to configure, but hard to trust.<n>This paper introduces the notion of a "safe passage": context generated by the LLM that is both grounded in the document and consistent with the extracted information.<n>Using these three parts in conjunction reduces hallucinations by up to 85% on information extraction tasks with minimal risk of flagging non-hallucinations.
  arXiv  Detail & Related papers  (2025-09-30T20:56:17Z)
• Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs [71.7892165868749]
  Commercial Large Language Model (LLM) APIs create a fundamental trust problem.<n>Users pay for specific models but have no guarantee that providers deliver them faithfully.<n>We formalize this model substitution problem and evaluate detection methods under realistic adversarial conditions.<n>We propose and evaluate the use of Trusted Execution Environments (TEEs) as one practical and robust solution.
  arXiv  Detail & Related papers  (2025-04-07T03:57:41Z)
• Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributions [60.43398881149664]
  We introduce LOS-Net, a lightweight attention-based architecture trained on an efficient encoding of the LLM Output Signature.<n>It achieves superior performance across diverse benchmarks and LLMs, while maintaining extremely low detection latency.
  arXiv  Detail & Related papers  (2025-03-18T09:04:37Z)
• Measuring memorization in language models via probabilistic extraction [29.438509661725117]
  Large language models (LLMs) are susceptible to memorizing training data.<n>Discoverable extraction is the most common method for measuring this issue.<n>We introduce probabilistic discoverable extraction, which, without additional cost, relaxes discoverable extraction by considering multiple queries.
  arXiv  Detail & Related papers  (2024-10-25T11:37:04Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? [60.50127555651554]
  Large Language Models (LLMs) show impressive results in numerous practical applications, but they lack essential safety features.<n>This makes them vulnerable to manipulations such as indirect prompt injections and generally unsuitable for safety-critical tasks.<n>We introduce a formal measure for instruction-data separation and an empirical variant that is calculable from a model's outputs.
  arXiv  Detail & Related papers  (2024-03-11T15:48:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.