Extractable Memorization From First Principles

• URL: http://arxiv.org/abs/2607.12649v1
• Date: Tue, 14 Jul 2026 11:31:36 GMT
• Title: Extractable Memorization From First Principles
• Abstract summary: We show that matched comparisons enable rigorous, calibrated memorization claims.<n>We refine "extractable memorization" to require a valid memorization claim and near-certain generation.
• Score: 66.10415626674103
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work on extractable memorization in LLMs suffers from two contrasting validity problems. Some studies overstate extraction, e.g., relying on sequences too short to distinguish memorization from predictability. Others imply that extraction is unreliable evidence of memorization, since models can also reproduce real-world text they weren't explicitly trained on. In different ways, both overlook what makes a valid extraction claim: the model must generate a training sequence with high enough probability to indicate memorization. To determine what's high enough, one has to perform a matched comparison: measuring the generation probabilities of both the training sequences of interest and comparable non-training sequences. Because non-training sequences cannot have been memorized, their probabilities provide a baseline for predictability; a training sequence exceeding this baseline provides evidence of memorization. We formalize matched comparisons in two ways: (1) a conformal test that calibrates a threshold to a chosen FPR when training and non-training sequences are sampled from populations, and (2) a census that calibrates against a matched non-training document when the object is a single document (e.g., a book). We show that matched comparisons enable rigorous, calibrated memorization claims, and reveal where prior setups have validity issues. For instance, on Wikipedia OLMo 2 32B reproduces non-training 10-token suffixes roughly 24% as often as training ones: that share of the training generation rate reflects false positives, not memorization. For Llama 3.1 70B on books, the thresholds we calibrate are as low as 1e-27, supporting memorization claims for sequences that no feasible sampling budget would extract. Based on these results, we refine "extractable memorization" to require a valid memorization claim and near-certain generation within a realistic budget.

Related papers

• Prior Aware Memorization: An Efficient Metric for Distinguishing Memorization from Generalization in Large Language Models [9.915106680104268]
  Training data leakage from Large Language Models (LLMs) raises serious concerns related to privacy, security, and copyright compliance.<n>Existing approaches to measuring memorization often conflate these phenomena, labeling outputs as memorized even when they arise from generalization over common patterns.<n>This work introduces Prior-Aware Memorization, a theoretically grounded, lightweight and training-free criterion for identifying genuine memorization in LLMs.
  arXiv  Detail & Related papers  (2026-02-21T06:31:17Z)
• Continual Memorization of Factoids in Language Models [32.37538704232502]
  Recent studies have shown that fine-tuning for memorization may be ineffective in storing knowledge or may exacerbate hallucinations.<n>We introduce a setting where a model must memorize and retain a set of factoids through multiple stages of fine-tuning on subsequent datasets.<n>We show that LMs widely suffer from forgetting, especially when needing to memorize factoids in the second stage.
  arXiv  Detail & Related papers  (2024-11-11T17:56:15Z)
• The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorization [15.348047288817478]
  We analyze the text memorization behavior of large language models (LLMs) when subjected to nucleus sampling. An increase of the nucleus size reduces memorization only modestly. Even when models do not engage in "hard" memorization, they may still display "soft" memorization.
  arXiv  Detail & Related papers  (2024-08-29T08:30:33Z)
• Demystifying Verbatim Memorization in Large Language Models [67.49068128909349]
  Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications. We develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to memorize verbatim sequences, even for out-of-distribution sequences.
  arXiv  Detail & Related papers  (2024-07-25T07:10:31Z)
• ROME: Memorization Insights from Text, Logits and Representation [17.458840481902644]
  This paper proposes an innovative approach named ROME that bypasses direct processing of the training data. Specifically, we select datasets categorized into three distinct types -- context-independent, conventional, and factual. Our analysis then focuses on disparities between memorized and non-memorized samples by examining the logits and representations of generated texts.
  arXiv  Detail & Related papers  (2024-03-01T13:15:30Z)
• Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy [91.98116450958331]
  We argue that verbatim memorization definitions are too restrictive and fail to capture more subtle forms of memorization. Specifically, we design and implement an efficient defense that perfectly prevents all verbatim memorization. We conclude by discussing potential alternative definitions and why defining memorization is a difficult yet crucial open question for neural language models.
  arXiv  Detail & Related papers  (2022-10-31T17:57:55Z)
• Reducing Training Sample Memorization in GANs by Training with Memorization Rejection [80.0916819303573]
  We propose rejection memorization, a training scheme that rejects generated samples that are near-duplicates of training samples during training. Our scheme is simple, generic and can be directly applied to any GAN architecture.
  arXiv  Detail & Related papers  (2022-10-21T20:17:50Z)
• An Empirical Study of Memorization in NLP [8.293936347234126]
  We use three different NLP tasks to check if the long-tail theory holds. Experiments demonstrate that top-ranked memorized training instances are likely atypical. We develop an attribution method to better understand why a training instance is memorized.
  arXiv  Detail & Related papers  (2022-03-23T03:27:56Z)
• Counterfactual Memorization in Neural Language Models [91.8747020391287]
  Modern neural language models that are widely used in various NLP tasks risk memorizing sensitive information from their training data. An open question in previous studies of language model memorization is how to filter out "common" memorization. We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if a particular document is omitted during training.
  arXiv  Detail & Related papers  (2021-12-24T04:20:57Z)
• Exploring Memorization in Adversarial Training [58.38336773082818]
  We investigate the memorization effect in adversarial training (AT) for promoting a deeper understanding of capacity, convergence, generalization, and especially robust overfitting. We propose a new mitigation algorithm motivated by detailed memorization analyses.
  arXiv  Detail & Related papers  (2021-06-03T05:39:57Z)
• L2R2: Leveraging Ranking for Abductive Reasoning [65.40375542988416]
  The abductive natural language inference task ($alpha$NLI) is proposed to evaluate the abductive reasoning ability of a learning system. A novel $L2R2$ approach is proposed under the learning-to-rank framework. Experiments on the ART dataset reach the state-of-the-art in the public leaderboard.
  arXiv  Detail & Related papers  (2020-05-22T15:01:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.