Does Engram Do Memory Retrieval in Autoregressive Image Generation?

• URL: http://arxiv.org/abs/2605.13179v1
• Date: Wed, 13 May 2026 08:40:46 GMT
• Title: Does Engram Do Memory Retrieval in Autoregressive Image Generation?
• Abstract summary: We adapt the Engram module to vision with 2D spatial $n$-gram hashing, gated fusion, and KV-cache-compatible incremental inference.<n>A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic.<n>A donor-probe experiment shows that swapping the hash inputs for matched, adversarial, or random same-class exemplars produces statistically indistinguishable next-token distributions.
• Score: 50.31813662801864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Engram module -- a hash-keyed, O(1) associative memory injected into Transformer layers -- was recently shown to improve large language model pretraining, with the appealing interpretation that it provides a content-addressed shortcut to recurring local token patterns. We ask whether this interpretation transfers to autoregressive (AR) image generation, or whether the observed gains, if any, come from a different mechanism. We adapt the Engram module to vision with 2D spatial $n$-gram hashing, gated fusion, and KV-cache-compatible incremental inference, and inject it into a class-conditional AR generator trained on ImageNet 256x256. Across a sweep of backbone-to-memory budget ratios $ρ{\in}[0.17, 0.90]$, every Engram-augmented variant trails the pure AR baseline in FID, indicating that the module saves backbone FLOPs but does not, by itself, improve sample quality. We then probe how the module is used. A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic, yet a tiny constant gate (g=0.10) matches or beats the learned gate -- inconsistent with a heavily content-addressed recall mechanism. A donor-probe experiment shows that swapping the hash inputs for matched, adversarial, or random same-class exemplars produces statistically indistinguishable next-token distributions, while collapsing or randomising the table degrades them by two to three orders of magnitude. Finally, training a model from scratch with the entire memory table frozen to $\mathcal{N}(0, 1)$ noise costs only $Δ\text{FID}{=}0.10$ and actually raises Inception Score. Together, these findings indicate that the Engram in AR image generation behaves not as a content-addressed retriever but as a gated architectural side-pathway: a hash-keyed residual stream whose benefit is dominated by the pathway itself, with the learned table contributing only a small distributional refinement.

Related papers

• Logit Refiner: Improving Visual Autoregressive Models via Intra-Scale Dependency Modeling [21.906953897227442]
  Visual Autoregressive Models generate images through next-scale prediction.<n>Mean-field approximation discards spatial dependencies among same-scale tokens.<n>Logit Refiner restores intra-scale dependencies by sequentially sampling tokens conditioned on frozen backbone features.
  arXiv  Detail & Related papers  (2026-09-10T16:57:05Z)
• Tokenizer-Agnostic Engram Module [5.1296785679475105]
  Deepseek's Engram conditional memory module relies on token-level $N$-gram hashing for Engram embedding lookup.<n>We propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers.<n>This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance.
  arXiv  Detail & Related papers  (2026-07-31T06:32:15Z)
• TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models [3.914493928069677]
  TF-Engram is a train-free Engram system that constructs phrase-specific semantic memory offline from external corpora.<n>System evaluation shows that TF-Engram tables can be built with moderate offline cost, SSD-backed storage substantially reduces GPU memory demand, and predictive prefetching recovers much of the throughput loss caused by external memory access.
  arXiv  Detail & Related papers  (2026-07-08T13:19:52Z)
• Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards [52.42920996842378]
  Most unified large multimodal models (LMMs) that support both visual understanding and image generation rely on curated post-training supervision.<n>We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a solver that answers and evaluates them, and a Generator that synthesizes images.<n>For image generation, we design a multi-scale internal evaluation scheme that combines question-answer fidelity scoring with cycle-consistent captioning.
  arXiv  Detail & Related papers  (2026-06-25T17:59:57Z)
• Lngram: N-gram Conditional Memory in Latent Space [7.279732869251628]
  We propose a latent-space conditional memory module that learns discrete symbols directly from hidden states.<n>Lngram outperforms Transformer and Engram baselines in long-context language modeling.<n>Analyses with LogitLens and CKA suggest that Lngram enables prediction-relevant information to emerge earlier.
  arXiv  Detail & Related papers  (2026-05-24T04:59:11Z)
• Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory [65.39827296429527]
  Scaling conditional memory offers a promising way to increase language-model capacity.<n>Existing methods such as Engram learn large memory tables from scratch during pre-training.<n>We propose Memory Grafting, a conditional memory scaling method that utilizes frozen hidden states from a grafting model as conditional n-gram memory.
  arXiv  Detail & Related papers  (2026-05-20T09:35:04Z)
• Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation [21.427403915969872]
  We observe that the per-position conditional entropy of the training set decays so quickly along the sequence that, after a few positions, the conditional distribution becomes essentially deterministic.<n>To address this, we propose Variable Codebook Size Quantization (VCQ), where the codebook size $K_t$ grows monotonically along the sequence.<n>With a vanilla autoregressive Transformer and standard next-token prediction, a base version of VCQ reduces gFID w/oG from 27.98 to 14.80 on ImageNet.
  arXiv  Detail & Related papers  (2026-05-07T13:13:30Z)
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models [42.816060150754645]
  We introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic $N$-gram embedding for O(1) lookup.<n>We scale Engram to 27B parameters, achieving superior performance over a strictly iso- parameter and iso-FLOPs MoE baseline.<n>We envision conditional memory as an indispensable modeling primitive for next-generation sparse models.
  arXiv  Detail & Related papers  (2026-01-12T09:54:49Z)
• REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization [130.46612643194973]
  reAR is a simple training strategy introducing a token-wise regularization objective.<n>On ImageNet, it reduces gFID from 3.02 to 1.86 and improves IS to 316.9 using a standardization-based tokenizer.<n>When applied to advanced tokenizers, it achieves a gFID of 1.42 with only 177M parameters, matching the performance with larger state-of-the-art diffusion models (675M)
  arXiv  Detail & Related papers  (2025-10-06T02:48:13Z)
• Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models [92.18057318458528]
  Token-Shuffle is a novel method that reduces the number of image tokens in Transformer.<n>Our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis.<n>In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15.
  arXiv  Detail & Related papers  (2025-04-24T17:59:56Z)
• Compile Scene Graphs with Reinforcement Learning [69.36723767339001]
  Next-token prediction is the fundamental principle for training large language models (LLMs)<n>We introduce R1-SGG, a multimodal LLM (M-LLM) initially trained via supervised fine-tuning (SFT) on the scene graph dataset.<n>We design a set of graph-centric rewards, including three recall-based variants -- Hard Recall, Hard Recall+Relax, and Soft Recall.
  arXiv  Detail & Related papers  (2025-04-18T10:46:22Z)
• Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation [34.112157859384645]
  Autoregressive (AR) modeling underpins state-of-the-art language and visual generative models.<n>Traditionally, a token'' is treated as the smallest prediction unit, often a discrete symbol in language or a quantized patch in vision.<n>We propose xAR, a framework that extends the notion of a token to an entity X.
  arXiv  Detail & Related papers  (2025-02-27T18:59:08Z)
• Spot the Error: Non-autoregressive Graphic Layout Generation with Wireframe Locator [37.65937921431886]
  We propose a learning-based locator to detect erroneous tokens which takes the wireframe image rendered from the generated layout sequence as input. We show that it serves as a complementary modality to the element sequence in object space and contributes greatly to the overall performance.
  arXiv  Detail & Related papers  (2024-01-29T18:13:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.