Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications

• URL: http://arxiv.org/abs/2407.19262v1
• Date: Sat, 27 Jul 2024 14:00:21 GMT
• Title: Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications
• Authors: Till Speicher, Mohammad Aflah Khan, Qinyuan Wu, Vedant Nanda, Soumi Das, Bishwamittra Ghosh, Krishna P. Gummadi, Evimaria Terzi,
• Abstract summary: We study whether and to what extent large language models (LLMs) have memorised training data. We create an experimental framework that is based on repeatedly exposing LLMs to random strings. We identify factors that make some strings easier to memorise than others, and we identify the role of local prefixes and global context in memorisation.
• Score: 14.818820873377303
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding whether and to what extent large language models (LLMs) have memorised training data has important implications for the reliability of their output and the privacy of their training data. In order to cleanly measure and disentangle memorisation from other phenomena (e.g. in-context learning), we create an experimental framework that is based on repeatedly exposing LLMs to random strings. Our framework allows us to better understand the dynamics, i.e., the behaviour of the model, when repeatedly exposing it to random strings. Using our framework, we make several striking observations: (a) we find consistent phases of the dynamics across families of models (Pythia, Phi and Llama2), (b) we identify factors that make some strings easier to memorise than others, and (c) we identify the role of local prefixes and global context in memorisation. We also show that sequential exposition to different random strings has a significant effect on memorisation. Our results, often surprising, have significant downstream implications in the study and usage of LLMs.

Related papers

• Dynamic Uncertainty Ranking: Enhancing In-Context Learning for Long-Tail Knowledge in LLMs [50.29035873837]
  Large language models (LLMs) can learn vast amounts of knowledge from diverse domains during pre-training. Long-tail knowledge from specialized domains is often scarce and underrepresented, rarely appearing in the models' memorization. We propose a reinforcement learning-based dynamic uncertainty ranking method for ICL that accounts for the varying impact of each retrieved sample on LLM predictions.
  arXiv  Detail & Related papers  (2024-10-31T03:42:17Z)
• Undesirable Memorization in Large Language Models: A Survey [5.659933808910005]
  We present a Systematization of Knowledge (SoK) on the topic of memorization in Large Language Models (LLMs) Memorization is the effect that a model tends to store and reproduce phrases or passages from the training data. We discuss the metrics and methods used to measure memorization, followed by an analysis of the factors that contribute to memorization phenomenon.
  arXiv  Detail & Related papers  (2024-10-03T16:34:46Z)
• Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Style [13.968658352075334]
  We investigate the impact of memory strength and evidence presentation on Large Language Models' receptiveness to external evidence. For questions with high memory strength, LLMs are more likely to rely on internal memory, particularly for larger LLMs such as GPT-4.
  arXiv  Detail & Related papers  (2024-09-17T07:44:06Z)
• Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data [76.90128359866462]
  We introduce an extended concept of memorization, distributional memorization, which measures the correlation between the output probabilities and the pretraining data frequency. This study demonstrates that memorization plays a larger role in simpler, knowledge-intensive tasks, while generalization is the key for harder, reasoning-based tasks.
  arXiv  Detail & Related papers  (2024-07-20T21:24:40Z)
• Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs [101.51435599249234]
  We propose an axiomatic system to define and quantify the precise memorization and in-context reasoning effects used by the large language model (LLM) Specifically, the axiomatic system enables us to categorize the memorization effects into foundational memorization effects and chaotic memorization effects. Experiments show that the clear disentanglement of memorization effects and in-context reasoning effects enables a straightforward examination of detailed inference patterns encoded by LLMs.
  arXiv  Detail & Related papers  (2024-05-20T08:51:03Z)
• A Multi-Perspective Analysis of Memorization in Large Language Models [10.276594755936529]
  Large Language Models (LLMs) show unprecedented performance in various fields. LLMs can generate the same content used to train them. This research comprehensively discussed memorization from various perspectives.
  arXiv  Detail & Related papers  (2024-05-19T15:00:50Z)
• LLM In-Context Recall is Prompt Dependent [0.0]
  A model's ability to do this significantly influences its practical efficacy and dependability in real-world applications. This study demonstrates that an LLM's recall capability is not only contingent upon the prompt's content but also may be compromised by biases in its training data.
  arXiv  Detail & Related papers  (2024-04-13T01:13:59Z)
• Exploring Memorization in Fine-tuned Language Models [53.52403444655213]
  We conduct the first comprehensive analysis to explore language models' memorization during fine-tuning across tasks. Our studies with open-sourced and our own fine-tuned LMs across various tasks indicate that memorization presents a strong disparity among different fine-tuning tasks. We provide an intuitive explanation of this task disparity via sparse coding theory and unveil a strong correlation between memorization and attention score distribution.
  arXiv  Detail & Related papers  (2023-10-10T15:41:26Z)
• Explaining Emergent In-Context Learning as Kernel Regression [61.57151500616111]
  Large language models (LLMs) have initiated a paradigm shift in transfer learning. In this paper, we investigate the reason why a transformer-based language model can accomplish in-context learning after pre-training. We find that during ICL, the attention and hidden features in LLMs match the behaviors of a kernel regression.
  arXiv  Detail & Related papers  (2023-05-22T06:45:02Z)
• Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning [104.58874584354787]
  In recent years, pre-trained large language models (LLMs) have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning. This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models.
  arXiv  Detail & Related papers  (2023-01-27T18:59:01Z)
• Understanding Unintended Memorization in Federated Learning [5.32880378510767]
  We show that different components of Federated Learning play an important role in reducing unintended memorization. We also show that training with a strong user-level differential privacy guarantee results in models that exhibit the least amount of unintended memorization.
  arXiv  Detail & Related papers  (2020-06-12T22:10:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.