A Fictional Q&A Dataset for Studying Memorization and Knowledge Acquisition

• URL: http://arxiv.org/abs/2506.05639v1
• Date: Thu, 05 Jun 2025 23:58:20 GMT
• Title: A Fictional Q&A Dataset for Studying Memorization and Knowledge Acquisition
• Authors: John Kirchenbauer, Janny Mongkolsupawan, Yuxin Wen, Tom Goldstein, Daphne Ippolito,
• Abstract summary: We propose a new dataset to study the dual processes of fact memorization and verbatim sequence memorization.<n>The dataset consists of synthetically-generated, webtext-like documents about fictional events, as well as question-answer pairs about the events.<n>We conduct training experiments showing how synthetic data about fictional events can be effective in teasing apart different forms of memorization.
• Score: 69.6105757233119
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When language models are trained on textual data, they acquire both knowledge about the structure of language as well as knowledge of facts about the world. At inference time, their knowledge of facts can be leveraged to solve interesting problems and perform useful knowledge work for users. It is well known that language models can verbatim memorize long sequences from their training data. However, it is much less well understood how language models memorize facts seen during training. In this work, we propose a new dataset to specifically empower researchers to study the dual processes of fact memorization and verbatim sequence memorization. The dataset consists of synthetically-generated, webtext-like documents about fictional events, as well as question-answer pairs about the events. We conduct training experiments showing how synthetic data about fictional events can be effective in teasing apart different forms of memorization. We also document the challenges in effectively building realistic, fictional synthetic data.

Related papers

• How do language models learn facts? Dynamics, curricula and hallucinations [22.693703460345873]
  Large language models accumulate vast knowledge during pre-training, yet the dynamics governing this acquisition remain poorly understood.<n>This work investigates the learning dynamics of language models on a synthetic factual recall task.
  arXiv  Detail & Related papers  (2025-03-27T16:43:45Z)
• Learning and Unlearning of Fabricated Knowledge in Language Models [16.971082623826263]
  We show that facts that conflict with common knowledge are remembered for tens of thousands of training steps. We show that impacts of knowledge-conflicting facts in LMs, though they can be long lasting, can be largely erased by novel application of multi-step sparse updates.
  arXiv  Detail & Related papers  (2024-10-29T05:33:14Z)
• Co-occurrence is not Factual Association in Language Models [19.708303468664088]
  We show that language models are biased to learn word co-occurrence statistics instead of true factual associations. We propose two strategies to improve the learning of factual associations in language models.
  arXiv  Detail & Related papers  (2024-09-21T08:13:16Z)
• Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data [76.90128359866462]
  We introduce an extended concept of memorization, distributional memorization, which measures the correlation between the output probabilities and the pretraining data frequency.<n>This study demonstrates that memorization plays a larger role in simpler, knowledge-intensive tasks, while generalization is the key for harder, reasoning-based tasks.
  arXiv  Detail & Related papers  (2024-07-20T21:24:40Z)
• Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research [139.69207791947738]
  Dolma is a three-trillion-token English corpus built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices.
  arXiv  Detail & Related papers  (2024-01-31T20:29:50Z)
• Blending Reward Functions via Few Expert Demonstrations for Faithful and Accurate Knowledge-Grounded Dialogue Generation [22.38338205905379]
  We leverage reinforcement learning algorithms to overcome the above challenges by introducing a novel reward function. Our reward function combines an accuracy metric and a faithfulness metric to provide a balanced quality judgment of generated responses.
  arXiv  Detail & Related papers  (2023-11-02T02:42:41Z)
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction [51.68385617116854]
  Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data.
  arXiv  Detail & Related papers  (2023-09-25T17:37:20Z)
• Explaining Patterns in Data with Language Models via Interpretable Autoprompting [143.4162028260874]
  We introduce interpretable autoprompting (iPrompt), an algorithm that generates a natural-language string explaining the data. iPrompt can yield meaningful insights by accurately finding groundtruth dataset descriptions. Experiments with an fMRI dataset show the potential for iPrompt to aid in scientific discovery.
  arXiv  Detail & Related papers  (2022-10-04T18:32:14Z)
• Counterfactual Memorization in Neural Language Models [91.8747020391287]
  Modern neural language models that are widely used in various NLP tasks risk memorizing sensitive information from their training data. An open question in previous studies of language model memorization is how to filter out "common" memorization. We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if a particular document is omitted during training.
  arXiv  Detail & Related papers  (2021-12-24T04:20:57Z)
• Probing Across Time: What Does RoBERTa Know and When? [70.20775905353794]
  We show that linguistic knowledge is acquired fast, stably, and robustly across domains. Facts and commonsense are slower and more domain-sensitive. We believe that probing-across-time analyses can help researchers understand the complex, intermingled learning that these models undergo and guide us toward more efficient approaches that accomplish necessary learning faster.
  arXiv  Detail & Related papers  (2021-04-16T04:26:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.