MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers

• URL: http://arxiv.org/abs/2607.10034v1
• Date: Fri, 10 Jul 2026 23:22:48 GMT
• Title: MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
• Abstract summary: We develop a theoretical account of fact-storage in large language models (LLMs)<n>We show that our construction attains the same storage capacity scaling up to penalization factors depending on the embedding.<n>As a proof-of-concept, we show that fact-storings enable modular fact editing by swapping Transformers with a new one.
• Score: 38.3661862596903
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) store factual knowledge in their parameters. While recent work has shown that this knowledge resides in MLP layers, existing constructive and mechanistic interpretability models of fact-storage in LLMs fail to explain the surprising empirical phenomenon that they store facts at an information-theoretically optimal rate. In this work, we develop a theoretical account of this phenomenon. We develop the first Transformer-compatible fact-storing MLP closed-form construction that satisfies the following three properties empirically observed in LLMs: it (i) attains optimal fact storage scaling, (ii) handles arbitrary input/output geometries, and (iii) works inside Transformers. Key to our work is to analyze the decoding margin of MLPs, whereas prior work only studies MLP fact storage. Under isotropic embeddings, our construction achieves information-theoretically optimal storage capacity scaling and requires $10$-$104\times$ fewer parameters at matched fact count than prior constructions. For arbitrary key and value embeddings, we show that our construction attains the same storage capacity scaling, up to penalization factors depending on the embedding geometries. Moreover, we demonstrate that our constructed MLPs can be used within Transformer blocks for factual recall tasks at optimal capacity scaling, requiring $15$-$63\times$ fewer parameters at matched fact count than prior constructions. Finally, as a proof-of-concept, we show that fact-storing MLPs enable modular fact editing by swapping a Transformer's MLP with a new one.

Related papers

• Geometric Factual Recall in Transformers [57.48371649045765]
  A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring memorization counts that scale linearly with the number of facts.<n>We develop a theoretical and empirical account of an alternative, emphgeometric form of memorization in which learned embeddings encode structure directly.<n>In a controlled setting where a single-layer transformer must memorize random vectors from subjects to a shared attribute set, we prove that a logarithmic embedding dimension suffices.<n>We extend these results to the multi-hop setting, providing constructions with and without chain-of-thought that exhibit a provable
  arXiv  Detail & Related papers  (2026-05-12T17:22:22Z)
• Constructing Efficient Fact-Storing MLPs for Transformers [9.371973249870207]
  We build explicit weight constructions to build fact-storings in large language models.<n>We demonstrate a proof-of-concept application of fact-storings: modular fact editing on one-layer Transformers by textit entires at once.
  arXiv  Detail & Related papers  (2025-11-28T21:18:35Z)
• Equivalence of Context and Parameter Updates in Modern Transformer Blocks [8.364690240329411]
  Recent research has established that the impact of context in a vanilla transformer can be represented implicitly by forming a token-dependent, rank-1 patch to its weights.<n>We first demonstrate a precise, analytical solution for a Gemma-style transformer block, proving that the entire effect of a context can be perfectly mapped to rank-1 patches.<n>We then generalize this result, providing a constructive proof and algorithm for multi-layer models.
  arXiv  Detail & Related papers  (2025-11-22T01:17:15Z)
• Understanding Factual Recall in Transformers via Associative Memories [55.93756571457904]
  We show that shallow transformers can use a combination of associative memories to obtain near optimal storage capacity.<n>We show that a transformer with a single layer of self-attention followed by an parameters can obtain 100% accuracy on a factual recall task.
  arXiv  Detail & Related papers  (2024-12-09T14:48:14Z)
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models [79.01926242857613]
  Large language models (LLMs) are prone to hallucinations, generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts.
  arXiv  Detail & Related papers  (2023-09-07T17:45:31Z)
• Efficient Language Modeling with Sparse all-MLP [53.81435968051093]
  All-MLPs can match Transformers in language modeling, but still lag behind in downstream tasks. We propose sparse all-MLPs with mixture-of-experts (MoEs) in both feature and input (tokens) We evaluate its zero-shot in-context learning performance on six downstream tasks, and find that it surpasses Transformer-based MoEs and dense Transformers.
  arXiv  Detail & Related papers  (2022-03-14T04:32:19Z)
• AS-MLP: An Axial Shifted MLP Architecture for Vision [50.11765148947432]
  An Axial Shifted architecture (AS-MLP) is proposed in this paper. By axially shifting channels of the feature map, AS-MLP is able to obtain the information flow from different directions. With the proposed AS-MLP architecture, our model obtains 83.3% Top-1 accuracy with 88M parameters and 15.2 GFLOPs on the ImageNet-1K dataset.
  arXiv  Detail & Related papers  (2021-07-18T08:56:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.