Analyzing Memorization in Large Language Models through the Lens of Model Attribution

• URL: http://arxiv.org/abs/2501.05078v1
• Date: Thu, 09 Jan 2025 09:00:32 GMT
• Title: Analyzing Memorization in Large Language Models through the Lens of Model Attribution
• Authors: Tarun Ram Menta, Susmit Agrawal, Chirag Agarwal,
• Abstract summary: Large Language Models (LLMs) are prevalent in modern applications but often memorize training data, leading to privacy breaches and copyright issues.<n>We investigate memorization from an architectural lens by analyzing how attention modules at different layers impact its memorization and generalization.
• Score: 11.295483963637217
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are prevalent in modern applications but often memorize training data, leading to privacy breaches and copyright issues. Existing research has mainly focused on posthoc analyses, such as extracting memorized content or developing memorization metrics, without exploring the underlying architectural factors that contribute to memorization. In this work, we investigate memorization from an architectural lens by analyzing how attention modules at different layers impact its memorization and generalization performance. Using attribution techniques, we systematically intervene in the LLM architecture by bypassing attention modules at specific blocks while keeping other components like layer normalization and MLP transformations intact. We provide theorems analyzing our intervention mechanism from a mathematical view, bounding the difference in layer outputs with and without our attributions. Our theoretical and empirical analyses reveal that attention modules in deeper transformer blocks are primarily responsible for memorization, whereas earlier blocks are crucial for the models generalization and reasoning capabilities. We validate our findings through comprehensive experiments on different LLM families (Pythia and GPTNeo) and five benchmark datasets. Our insights offer a practical approach to mitigate memorization in LLMs while preserving their performance, contributing to safer and more ethical deployment in real world applications.

Related papers

• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks. We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Think or Remember? Detecting and Directing LLMs Towards Memorization or Generalization [10.11351265687309]
  We explore the foundational mechanisms of memorization and generalization in Large Language Models (LLMs) Our investigation serves as a case study leveraging specially designed datasets and experimental-scale LLMs.
  arXiv  Detail & Related papers  (2024-12-24T15:28:56Z)
• Detecting Memorization in Large Language Models [0.0]
  Large language models (LLMs) have achieved impressive results in natural language processing but are prone to memorizing portions of their training data. Traditional methods for detecting memorization rely on output probabilities or loss functions. We introduce an analytical method that precisely detects memorization by examining neuron activations within the LLM.
  arXiv  Detail & Related papers  (2024-12-02T00:17:43Z)
• Understanding Ranking LLMs: A Mechanistic Analysis for Information Retrieval [20.353393773305672]
  We employ a probing-based analysis to examine neuron activations in ranking LLMs. Our study spans a broad range of feature categories, including lexical signals, document structure, query-document interactions, and complex semantic representations. Our findings offer crucial insights for developing more transparent and reliable retrieval systems.
  arXiv  Detail & Related papers  (2024-10-24T08:20:10Z)
• Configurable Foundation Models: Building LLMs from a Modular Perspective [115.63847606634268]
  A growing tendency to decompose LLMs into numerous functional modules allows for inference with part of modules and dynamic assembly of modules to tackle complex tasks. We coin the term brick to represent each functional module, designating the modularized structure as customizable foundation models. We present four brick-oriented operations: retrieval and routing, merging, updating, and growing. We find that the FFN layers follow modular patterns with functional specialization of neurons and functional neuron partitions.
  arXiv  Detail & Related papers  (2024-09-04T17:01:02Z)
• A Survey on the Memory Mechanism of Large Language Model based Agents [66.4963345269611]
  Large language model (LLM) based agents have recently attracted much attention from the research and industry communities. LLM-based agents are featured in their self-evolving capability, which is the basis for solving real-world problems. The key component to support agent-environment interactions is the memory of the agents.
  arXiv  Detail & Related papers  (2024-04-21T01:49:46Z)
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension [63.330262740414646]
  We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs) We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
  arXiv  Detail & Related papers  (2024-02-28T04:56:21Z)
• LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
  Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
  arXiv  Detail & Related papers  (2024-02-26T07:33:05Z)
• Do LLMs Dream of Ontologies? [13.776194387957617]
  Large Models Language (LLMs) have demonstrated remarkable memorization across diverse natural language processing tasks. This paper investigates the extent to which general-purpose LLMs correctly reproduce concept identifier (ID)-label associations from publicly available resources.
  arXiv  Detail & Related papers  (2024-01-26T15:10:23Z)
• Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations [50.81844184210381]
  We propose a general paradigm that augments large language models with DOmain-specific KnowledgE to enhance their performance on practical applications, namely DOKE. This paradigm relies on a domain knowledge extractor, working in three steps: 1) preparing effective knowledge for the task; 2) selecting the knowledge for each specific sample; and 3) expressing the knowledge in an LLM-understandable way.
  arXiv  Detail & Related papers  (2023-11-16T07:09:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.