Dissecting Recall of Factual Associations in Auto-Regressive Language Models

• URL: http://arxiv.org/abs/2304.14767v3
• Date: Fri, 13 Oct 2023 19:01:20 GMT
• Title: Dissecting Recall of Factual Associations in Auto-Regressive Language Models
• Authors: Mor Geva, Jasmijn Bastings, Katja Filippova, Amir Globerson
• Abstract summary: Transformer-based language models (LMs) are known to capture factual knowledge in their parameters. We study how the model aggregates information about the subject and relation to predict the correct attribute. Our findings introduce a comprehensive view of how factual associations are stored and extracted internally in LMs.
• Score: 41.71388509750695
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based language models (LMs) are known to capture factual knowledge in their parameters. While previous work looked into where factual associations are stored, only little is known about how they are retrieved internally during inference. We investigate this question through the lens of information flow. Given a subject-relation query, we study how the model aggregates information about the subject and relation to predict the correct attribute. With interventions on attention edges, we first identify two critical points where information propagates to the prediction: one from the relation positions followed by another from the subject positions. Next, by analyzing the information at these points, we unveil a three-step internal mechanism for attribute extraction. First, the representation at the last-subject position goes through an enrichment process, driven by the early MLP sublayers, to encode many subject-related attributes. Second, information from the relation propagates to the prediction. Third, the prediction representation "queries" the enriched subject to extract the attribute. Perhaps surprisingly, this extraction is typically done via attention heads, which often encode subject-attribute mappings in their parameters. Overall, our findings introduce a comprehensive view of how factual associations are stored and extracted internally in LMs, facilitating future research on knowledge localization and editing.

Related papers

• Multiple Streams of Relation Extraction: Enriching and Recalling in Transformers [9.901842773988946]
  We show that fine-tuned language models both extract relation information learned during finetuning while processing entities and (2) recall" this information in later layers while generating predictions.<n>We examine the necessity and sufficiency of these information pathways, examining what layers they occur at, how much redundancy they exhibit, and which model components are involved.
  arXiv  Detail & Related papers  (2025-06-25T18:13:34Z)
• Knowledge Graph Completion with Relation-Aware Anchor Enhancement [50.50944396454757]
  We propose a relation-aware anchor enhanced knowledge graph completion method (RAA-KGC) We first generate anchor entities within the relation-aware neighborhood of the head entity. Then, by pulling the query embedding towards the neighborhoods of the anchors, it is tuned to be more discriminative for target entity matching.
  arXiv  Detail & Related papers  (2025-04-08T15:22:08Z)
• An Attempt to Unraveling Token Prediction Refinement and Identifying Essential Layers of Large Language Models [0.0]
  This research aims to unravel how large language models (LLMs) iteratively refine token predictions. We focused on how LLMs access and use information from input contexts, and how positioning of relevant information affects the model's token prediction refinement process.
  arXiv  Detail & Related papers  (2025-01-25T03:34:15Z)
• Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation [81.18701211912779]
  We introduce an Adaptive Multi-Aspect Retrieval-augmented over KGs (Amar) framework. This method retrieves knowledge including entities, relations, and subgraphs, and converts each piece of retrieved text into prompt embeddings. Our method has achieved state-of-the-art performance on two common datasets.
  arXiv  Detail & Related papers  (2024-12-24T16:38:04Z)
• Deciphering the Interplay of Parametric and Non-parametric Memory in Retrieval-augmented Language Models [5.274653527674298]
  Retrieval-Augmented Generation (RAG) models act like retrieving information before generating responses. We use causal mediation analysis and controlled experiments to examine how internal representations influence information processing.
  arXiv  Detail & Related papers  (2024-10-07T16:14:47Z)
• Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction [36.40833517478628]
  Large language models require updates to remain up-to-date or adapt to new domains. One key is memorizing the latest information in a way that the memorized information is extractable with a query prompt. Despite minimizing document perplexity during fine-tuning, LLMs struggle to extract information through a prompt sentence.
  arXiv  Detail & Related papers  (2024-02-16T06:29:16Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)
• MIANet: Aggregating Unbiased Instance and General Information for Few-Shot Semantic Segmentation [6.053853367809978]
  Existing few-shot segmentation methods are based on the meta-learning strategy and extract instance knowledge from a support set. We propose a multi-information aggregation network (MIANet) that effectively leverages the general knowledge, i.e., semantic word embeddings, and instance information for accurate segmentation. Experiments on PASCAL-5i and COCO-20i show that MIANet yields superior performance and set a new state-of-the-art.
  arXiv  Detail & Related papers  (2023-05-23T09:36:27Z)
• Link Prediction on N-ary Relational Data Based on Relatedness Evaluation [61.61555159755858]
  We propose a method called NaLP to conduct link prediction on n-ary relational data. We represent each n-ary relational fact as a set of its role and role-value pairs. Experimental results validate the effectiveness and merits of the proposed methods.
  arXiv  Detail & Related papers  (2021-04-21T09:06:54Z)
• Learning from Context or Names? An Empirical Study on Neural Relation Extraction [112.06614505580501]
  We study the effect of two main information sources in text: textual context and entity mentions (names) We propose an entity-masked contrastive pre-training framework for relation extraction (RE) Our framework can improve the effectiveness and robustness of neural models in different RE scenarios.
  arXiv  Detail & Related papers  (2020-10-05T11:21:59Z)
• Attention improves concentration when learning node embeddings [1.2233362977312945]
  Given nodes labelled with search query text, we want to predict links to related queries that share products. Experiments with a range of deep neural architectures show that simple feedforward networks with an attention mechanism perform best for learning embeddings. We propose an analytically tractable model of query generation, AttEST, that views both products and the query text as vectors embedded in a latent space.
  arXiv  Detail & Related papers  (2020-06-11T21:21:12Z)
• Visual Question Answering with Prior Class Semantics [50.845003775809836]
  We show how to exploit additional information pertaining to the semantics of candidate answers. We extend the answer prediction process with a regression objective in a semantic space. Our method brings improvements in consistency and accuracy over a range of question types.
  arXiv  Detail & Related papers  (2020-05-04T02:46:31Z)
• Self-Attention Attribution: Interpreting Information Interactions Inside Transformer [89.21584915290319]
  We propose a self-attention attribution method to interpret the information interactions inside Transformer. We show that the attribution results can be used as adversarial patterns to implement non-targeted attacks towards BERT.
  arXiv  Detail & Related papers  (2020-04-23T14:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.