Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs

• URL: http://arxiv.org/abs/2606.08347v1
• Date: Sat, 06 Jun 2026 21:36:51 GMT
• Title: Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
• Abstract summary: Over-tokenized Transformers and Engram attempt to address this limitation by explicitly incorporating multi-token memories.<n>We propose Polyadicized Engram (TN-gram), a compact memory module that represents n-gram embeddings through shared factors.<n> Comprehensive experiments demonstrate that TN-gram matches or even outperforms Engram-style n-gram modules.
• Score: 29.319221741796685
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern language models represent text using discrete token-level embeddings, which forces recurring multi-token patterns to be learned implicitly across Transformer layers. Both Over-tokenized Transformers and Engram attempt to address this limitation by explicitly incorporating multi-token (n-gram) memories. However, they rely on separate hash tables for each n-gram order, which introduces hash collisions and prevents nested n-grams from sharing the underlying latent structures. To address these issues, we propose Tensorized Engram (TN-gram), a compact memory module that represents tensorized n-gram embeddings through shared factors in the Canonical Polyadic (CP) form. TN-gram learns shared token-position factors together with order-absorption vectors to encode the embeddings of different n-gram order. Comprehensive experiments demonstrate that TN-gram matches or even outperforms Engram-style n-gram modules while requiring much fewer parameters.

Related papers

• Tokenizer-Agnostic Engram Module [5.1296785679475105]
  Deepseek's Engram conditional memory module relies on token-level $N$-gram hashing for Engram embedding lookup.<n>We propose a change to the hashing routine, enabling compatibility between Engram models using different tokenizers.<n>This work investigates the possible trade-offs and shows that this simple substitution produces comparable performance.
  arXiv  Detail & Related papers  (2026-07-31T06:32:15Z)
• Lngram: N-gram Conditional Memory in Latent Space [7.279732869251628]
  We propose a latent-space conditional memory module that learns discrete symbols directly from hidden states.<n>Lngram outperforms Transformer and Engram baselines in long-context language modeling.<n>Analyses with LogitLens and CKA suggest that Lngram enables prediction-relevant information to emerge earlier.
  arXiv  Detail & Related papers  (2026-05-24T04:59:11Z)
• Does Engram Do Memory Retrieval in Autoregressive Image Generation? [50.31813662801864]
  We adapt the Engram module to vision with 2D spatial $n$-gram hashing, gated fusion, and KV-cache-compatible incremental inference.<n>A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic.<n>A donor-probe experiment shows that swapping the hash inputs for matched, adversarial, or random same-class exemplars produces statistically indistinguishable next-token distributions.
  arXiv  Detail & Related papers  (2026-05-13T08:40:46Z)
• Enhanced Graph Transformer with Serialized Graph Tokens [28.343061516069756]
  Transformers have demonstrated success in graph learning, particularly for node-level tasks.<n>We propose a novel serialized token paradigm to encapsulate global signals more effectively.<n>We show that our method achieves state-of-the-art results on several graph-level benchmarks.
  arXiv  Detail & Related papers  (2026-02-09T07:23:22Z)
• Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens [138.36729703589512]
  We show that $n$-gram language models are still relevant in this era of neural large language models (LLMs)<n>This was done by modernizing $n$-gram LMs in two aspects. First, we train them at the same data scale as neural LLMs -- 5 trillion tokens.<n>Second, existing $n$-gram LMs use small $n$ which hinders their performance; we instead allow $n$ to be arbitrarily large, by introducing a new $infty$-gram LM with backoff.
  arXiv  Detail & Related papers  (2024-01-30T19:03:49Z)
• DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning [62.51232333352754]
  Text-to-image (T2I) generation has seen significant growth over the past few years. Despite this, there has been little work on generating diagrams with T2I models. We present DiagrammerGPT, a novel two-stage text-to-diagram generation framework. We show that our framework produces more accurate diagrams, outperforming existing T2I models.
  arXiv  Detail & Related papers  (2023-10-18T17:37:10Z)
• State space models can express n-gram languages [51.823427608117626]
  We build state space language models that can solve the next-word prediction task for languages generated from n-gram rules.<n>Our proof shows how SSMs can encode n-gram rules using new theoretical results on their capacity.<n>We conduct experiments with a small dataset generated from n-gram rules to show how our framework can be applied to SSMs and RNNs obtained through gradient-based optimization.
  arXiv  Detail & Related papers  (2023-06-20T10:41:23Z)
• Discrete Graph Auto-Encoder [52.50288418639075]
  We introduce a new framework named Discrete Graph Auto-Encoder (DGAE) We first use a permutation-equivariant auto-encoder to convert graphs into sets of discrete latent node representations. In the second step, we sort the sets of discrete latent representations and learn their distribution with a specifically designed auto-regressive model.
  arXiv  Detail & Related papers  (2023-06-13T12:40:39Z)
• Transforming Visual Scene Graphs to Image Captions [69.13204024990672]
  We propose to transform Scene Graphs (TSG) into more descriptive captions. In TSG, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. In TSG, each expert is built on MHA, for discriminating the graph embeddings to generate different kinds of words.
  arXiv  Detail & Related papers  (2023-05-03T15:18:37Z)
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization [50.16128796194463]
  We propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. We introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level.
  arXiv  Detail & Related papers  (2021-06-23T22:24:14Z)
• NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs [15.289356276538662]
  We propose NodePiece, an anchor-based approach to learn a fixed-size entity vocabulary. In NodePiece, a vocabulary of subword/sub-entity units is constructed from anchor nodes in a graph with known relation types. Experiments show that NodePiece performs competitively in node classification, link prediction, and relation prediction tasks.
  arXiv  Detail & Related papers  (2021-06-23T03:51:03Z)
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding [44.21363470798758]
  We propose ERNIE-Gram, an explicitly n-gram masking method to enhance the integration of coarse-grained information into pre-training. We pre-train ERNIE-Gram on English and Chinese text corpora and fine-tune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin.
  arXiv  Detail & Related papers  (2020-10-23T03:42:20Z)
• Neural Machine Translation without Embeddings [44.129310924201604]
  Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes via-8. Experiments on byteto-byte machine translation from English to 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models.
  arXiv  Detail & Related papers  (2020-08-21T09:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.