Tokenizer-Agnostic Engram Module

• URL: http://arxiv.org/abs/2607.29065v1
• Date: Fri, 31 Jul 2026 06:32:15 GMT
• Title: Tokenizer-Agnostic Engram Module
• Abstract summary: Deepseek's Engram conditional memory module relies on token-level $N$-gram hashing for Engram embedding lookup.<n>We propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers.<n>This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance.
• Score: 5.1296785679475105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deepseek's Engram, a conditional memory module, was introduced to trade-off storage versus reasoning in large language models. However, the module relies on token-level $N$-gram hashing for Engram embedding lookup, introducing a tight coupling to the tokenizer used: a model with a different tokenizer would have to train its own Engram embeddings from scratch. To improve the reusability of Engram embeddings, we propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers. Instead of modelling disjoint $N$-gram spaces, we treat $N$-gram as a method to sample potentially useful byte sequences, from all possible byte sequences across tokens. We replace the XOR-based hashing with the general polynomial hashing with a joint embedding space across $N$. This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance and achieves tokenizer-agnosticism: hash equivalence for byte-equivalent token sequences.

Related papers

• MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment [0.0]
  We introduce MinGram, which keeps the token-list representation but simplifies training using a BPE-derived seed vocabulary.<n>In controlled downstream language-model training, Unigram-family tokenizers, with MinGram among the best, consistently beat BPE in bits-per-byte.
  arXiv  Detail & Related papers  (2026-06-25T13:31:02Z)
• Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs [29.319221741796685]
  Over-tokenized Transformers and Engram attempt to address this limitation by explicitly incorporating multi-token memories.<n>We propose Polyadicized Engram (TN-gram), a compact memory module that represents n-gram embeddings through shared factors.<n> Comprehensive experiments demonstrate that TN-gram matches or even outperforms Engram-style n-gram modules.
  arXiv  Detail & Related papers  (2026-06-06T21:36:51Z)
• Lngram: N-gram Conditional Memory in Latent Space [7.279732869251628]
  We propose a latent-space conditional memory module that learns discrete symbols directly from hidden states.<n>Lngram outperforms Transformer and Engram baselines in long-context language modeling.<n>Analyses with LogitLens and CKA suggest that Lngram enables prediction-relevant information to emerge earlier.
  arXiv  Detail & Related papers  (2026-05-24T04:59:11Z)
• Does Engram Do Memory Retrieval in Autoregressive Image Generation? [50.31813662801864]
  We adapt the Engram module to vision with 2D spatial $n$-gram hashing, gated fusion, and KV-cache-compatible incremental inference.<n>A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic.<n>A donor-probe experiment shows that swapping the hash inputs for matched, adversarial, or random same-class exemplars produces statistically indistinguishable next-token distributions.
  arXiv  Detail & Related papers  (2026-05-13T08:40:46Z)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations [83.93566096400723]
  We find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization.<n>Character-level segmentation improves string manipulation and code understanding tasks by up to +14%.<n>Right-aligned digit grouping enhances large-number arithmetic by +33%.
  arXiv  Detail & Related papers  (2025-06-23T18:02:26Z)
• Sampling from Your Language Model One Byte at a Time [82.71473348639489]
  Tokenization can introduce distortion into the model's generations, known as the Prompt Boundary Problem (PBP)<n>We present an inference-time method to convert any autore LM with a BPE tokenizer into a character-level or byte-level LM.<n>Our method efficiently solves the PBP and is also able to unify the vocabularies of language models with different tokenizers.
  arXiv  Detail & Related papers  (2025-06-17T02:37:04Z)
• Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit [45.18582668677648]
  We present a training-free method to transplant tokenizers in large language models.<n>We approximate each out-of-vocabulary token as a sparse linear combination of shared tokens.<n>We show that OMP achieves best zero-shot preservation of the base model's performance.
  arXiv  Detail & Related papers  (2025-06-07T00:51:27Z)
• Causal Estimation of Tokenisation Bias [58.20086589761273]
  We quantify the effect of including or not a subword in a tokeniser's vocabulary on the probability a trained model assigns to the corresponding characters.<n>We find that tokenisation consistently affects models' outputs across scales, vocabularies, and tokenisers.<n> Notably, a subword's presence in a small model's vocabulary may increase its characters' probability by up to 17 times.
  arXiv  Detail & Related papers  (2025-06-03T17:59:47Z)
• CODA: Repurposing Continuous VAEs for Discrete Tokenization [31.932323809073477]
  textbfCODA(textbfCOntinuous-to-textbfDiscrete textbfAdaptation) is a framework that decouples compression and discretization.<n>Our approach achieves a remarkable codebook utilization of 100% and notable reconstruction FID (rFID) of $mathbf0.43$ and $mathbf1.34$ for $8 times$ and $16 times$ compression on ImageNet 256$times$ 256 benchmark.
  arXiv  Detail & Related papers  (2025-03-22T12:59:00Z)
• Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens [138.36729703589512]
  We show that $n$-gram language models are still relevant in this era of neural large language models (LLMs)<n>This was done by modernizing $n$-gram LMs in two aspects. First, we train them at the same data scale as neural LLMs -- 5 trillion tokens.<n>Second, existing $n$-gram LMs use small $n$ which hinders their performance; we instead allow $n$ to be arbitrarily large, by introducing a new $infty$-gram LM with backoff.
  arXiv  Detail & Related papers  (2024-01-30T19:03:49Z)
• Object Recognition as Next Token Prediction [99.40793702627396]
  We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels.
  arXiv  Detail & Related papers  (2023-12-04T18:58:40Z)
• byteSteady: Fast Classification Using Byte-Level n-Gram Embeddings [77.6701264226519]
  We introduce byteSteady, a fast model for classification using byte-level n-gram embeddings. A straightforward application of byteSteady is text classification. We also apply byteSteady to one type of non-language data -- DNA sequences for gene classification.
  arXiv  Detail & Related papers  (2021-06-24T20:14:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.