Geometric Factual Recall in Transformers

• URL: http://arxiv.org/abs/2605.12426v1
• Date: Tue, 12 May 2026 17:22:22 GMT
• Title: Geometric Factual Recall in Transformers
• Abstract summary: A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring memorization counts that scale linearly with the number of facts.<n>We develop a theoretical and empirical account of an alternative, emphgeometric form of memorization in which learned embeddings encode structure directly.<n>In a controlled setting where a single-layer transformer must memorize random vectors from subjects to a shared attribute set, we prove that a logarithmic embedding dimension suffices.<n>We extend these results to the multi-hop setting, providing constructions with and without chain-of-thought that exhibit a provable
• Score: 57.48371649045765
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How do transformer language models memorize factual associations? A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, \emph{geometric} form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer must memorize random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension suffices: subject embeddings encode \emph{linear superpositions} of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute via ReLU gating, and not as an associative key-value mapping. We extend these results to the multi-hop setting -- chains of relational queries such as ``Who is the mother of the wife of $x$?'' -- providing constructions with and without chain-of-thought that exhibit a provable capacity-depth tradeoff, complemented by a matching information-theoretic lower bound. Empirically, gradient descent discovers solutions with precisely the predicted structure. Once trained, the MLP transfers zero-shot to entirely new bijections when subject embeddings are appropriately re-initialized, revealing that it has learned a generic selection mechanism rather than memorized any particular set of facts.

Related papers

• A Compositional Theory of Causally Masked Transformers [50.88598486616582]
  We develop a formalization that derives expressivity directly from the model's implemented dynamics.<n>Each attention head updates its own state independently within a layer.<n>A modified form of soft attention supports irreversible, checklist-like state.
  arXiv  Detail & Related papers  (2026-07-29T14:47:19Z)
• Fine-Tuning Dynamics of In-Context Factual Recall in Transformers [63.1821972855273]
  In-context learning -- performing tasks based on examples given in the prompt -- has emerged in large language models.<n>We study how incontext learning leverages factual knowledge recall.<n>We analyze the supervised fine-tuning dynamics of a one-layer transformer on IC-recall data.
  arXiv  Detail & Related papers  (2026-05-26T23:47:26Z)
• Knowledge Graphs as Structured Memory for Embedding Spaces: From Training Clusters to Explainable Inference [3.2945446636945963]
  Graph Memory (GM) is a structured non-parametric framework that augments embedding-based inference with a compact, relational memory over region-level prototypes.<n>By explicitly modeling reliability and relational structure, GM provides a principled bridge between local evidence and global consistency in non-parametric learning.
  arXiv  Detail & Related papers  (2025-11-18T23:02:59Z)
• Understanding Factual Recall in Transformers via Associative Memories [55.93756571457904]
  We show that shallow transformers can use a combination of associative memories to obtain near optimal storage capacity.<n>We show that a transformer with a single layer of self-attention followed by an parameters can obtain 100% accuracy on a factual recall task.
  arXiv  Detail & Related papers  (2024-12-09T14:48:14Z)
• Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing [10.206921909332006]
  Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate.<n>We discover that the parameter initialization scale plays a critical role in determining whether the model learns inferential (reasoning-based) solutions.<n>We further find that inferential (reasoning-based) solutions exhibit low complexity bias, which we hypothesize is a key factor enabling them to learn individual mappings for single anchors.
  arXiv  Detail & Related papers  (2024-05-08T20:23:24Z)
• Linearity of Relation Decoding in Transformer Language Models [82.47019600662874]
  Much of the knowledge encoded in transformer language models (LMs) may be expressed in terms of relations. We show that, for a subset of relations, this computation is well-approximated by a single linear transformation on the subject representation.
  arXiv  Detail & Related papers  (2023-08-17T17:59:19Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.