Lngram: N-gram Conditional Memory in Latent Space

• URL: http://arxiv.org/abs/2605.24869v1
• Date: Sun, 24 May 2026 04:59:11 GMT
• Title: Lngram: N-gram Conditional Memory in Latent Space
• Abstract summary: We propose a latent-space conditional memory module that learns discrete symbols directly from hidden states.<n>Lngram outperforms Transformer and Engram baselines in long-context language modeling.<n>Analyses with LogitLens and CKA suggest that Lngram enables prediction-relevant information to emerge earlier.
• Score: 7.279732869251628
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sequence modeling requires both compositional reasoning and local static knowledge retrieval, yet standard Transformers handle both through dense computation. Engram partially decouples retrieval from the backbone, but its token-based keys remain tied to text tokenization and hash compression. We propose Lngram, a latent-space conditional memory module that learns discrete symbols directly from hidden states and performs N-gram lookup over these symbols. This design removes the dependence on tokenizer IDs and naturally extends to non-text modalities. In our evaluated settings, Lngram outperforms Transformer and Engram baselines, consistently reduces perplexity in long-context language modeling, and effectively injects domain knowledge when added post hoc to pretrained models. Joint training with the backbone further surpasses full fine-tuning, while experiments on vision-language and vision-language-action tasks show overall gains. Analyses with LogitLens and CKA suggest that Lngram enables prediction-relevant information to emerge earlier, increasing effective depth with limited inference and memory overhead. Code is available at https://github.com/zyaaa-ux/Lngram.

Related papers

• Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs [29.319221741796685]
  Over-tokenized Transformers and Engram attempt to address this limitation by explicitly incorporating multi-token memories.<n>We propose Polyadicized Engram (TN-gram), a compact memory module that represents n-gram embeddings through shared factors.<n> Comprehensive experiments demonstrate that TN-gram matches or even outperforms Engram-style n-gram modules.
  arXiv  Detail & Related papers  (2026-06-06T21:36:51Z)
• Does Engram Do Memory Retrieval in Autoregressive Image Generation? [50.31813662801864]
  We adapt the Engram module to vision with 2D spatial $n$-gram hashing, gated fusion, and KV-cache-compatible incremental inference.<n>A gate-clamp sweep shows that disabling the Engram pathway entirely is catastrophic.<n>A donor-probe experiment shows that swapping the hash inputs for matched, adversarial, or random same-class exemplars produces statistically indistinguishable next-token distributions.
  arXiv  Detail & Related papers  (2026-05-13T08:40:46Z)
• Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models [32.05060138278358]
  We probe the internal representation of large vision-language models (LVLMs) using a synthetic diagram dataset based on directed graphs.<n>Our experiments reveal that edge information is not linearly separable in the vision encoder and becomes linearly encoded only in the text tokens in the language model.<n>These findings suggest that the stage at which linearly separable representations are formed varies depending on the type of visual information.
  arXiv  Detail & Related papers  (2026-03-03T11:17:31Z)
• Semi-supervised Instruction Tuning for Large Language Models on Text-Attributed Graphs [62.544129365882014]
  We propose a novel Semi-supervised Instruction Tuning pipeline for Graph Learning, named SIT-Graph.<n> SIT-Graph is model-agnostic and can be seamlessly integrated into any graph instruction tuning method that utilizes LLMs as the predictor.<n>Extensive experiments demonstrate that when incorporated into state-of-the-art graph instruction tuning methods, SIT-Graph significantly enhances their performance on text-attributed graph benchmarks.
  arXiv  Detail & Related papers  (2026-01-19T08:10:53Z)
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models [42.816060150754645]
  We introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic $N$-gram embedding for O(1) lookup.<n>We scale Engram to 27B parameters, achieving superior performance over a strictly iso- parameter and iso-FLOPs MoE baseline.<n>We envision conditional memory as an indispensable modeling primitive for next-generation sparse models.
  arXiv  Detail & Related papers  (2026-01-12T09:54:49Z)
• Semantic-Condition Tuning: Fusing Graph Context with Large Language Models for Knowledge Graph Completion [6.895510903358326]
  Fusing Knowledge Graphs with Large Language Models is crucial for knowledge-intensive tasks like knowledge graph completion.<n>We propose Semantic-Adaptive Graph Tuning (SCT), a new knowledge injection paradigm comprising two key modules.<n>SCT provides a more direct and potent signal, enabling more accurate and robust knowledge reasoning.
  arXiv  Detail & Related papers  (2025-10-10T03:22:27Z)
• Align-GRAG: Reasoning-Guided Dual Alignment for Graph Retrieval-Augmented Generation [79.75818239774952]
  Large language models (LLMs) have demonstrated remarkable capabilities, but still struggle with issues like hallucinations and outdated information.<n>Retrieval-augmented generation (RAG) addresses these issues by grounding LLM outputs in external knowledge with an Information Retrieval (IR) system.<n>We propose Align-GRAG, a novel reasoning-guided dual alignment framework in post-retrieval phrase.
  arXiv  Detail & Related papers  (2025-05-22T05:15:27Z)
• Compile Scene Graphs with Reinforcement Learning [69.36723767339001]
  Next-token prediction is the fundamental principle for training large language models (LLMs)<n>We introduce R1-SGG, a multimodal LLM (M-LLM) initially trained via supervised fine-tuning (SFT) on the scene graph dataset.<n>We design a set of graph-centric rewards, including three recall-based variants -- Hard Recall, Hard Recall+Relax, and Soft Recall.
  arXiv  Detail & Related papers  (2025-04-18T10:46:22Z)
• Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens [138.36729703589512]
  We show that $n$-gram language models are still relevant in this era of neural large language models (LLMs)<n>This was done by modernizing $n$-gram LMs in two aspects. First, we train them at the same data scale as neural LLMs -- 5 trillion tokens.<n>Second, existing $n$-gram LMs use small $n$ which hinders their performance; we instead allow $n$ to be arbitrarily large, by introducing a new $infty$-gram LM with backoff.
  arXiv  Detail & Related papers  (2024-01-30T19:03:49Z)
• State space models can express n-gram languages [51.823427608117626]
  We build state space language models that can solve the next-word prediction task for languages generated from n-gram rules.<n>Our proof shows how SSMs can encode n-gram rules using new theoretical results on their capacity.<n>We conduct experiments with a small dataset generated from n-gram rules to show how our framework can be applied to SSMs and RNNs obtained through gradient-based optimization.
  arXiv  Detail & Related papers  (2023-06-20T10:41:23Z)
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding [44.21363470798758]
  We propose ERNIE-Gram, an explicitly n-gram masking method to enhance the integration of coarse-grained information into pre-training. We pre-train ERNIE-Gram on English and Chinese text corpora and fine-tune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin.
  arXiv  Detail & Related papers  (2020-10-23T03:42:20Z)
• My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control [65.77164390203396]
  We present a series of ablations on existing methods that show that morphological information encoded in the graph does not improve their performance. Motivated by the hypothesis that any benefits GNNs extract from the graph structure are outweighed by difficulties they create for message passing, we also propose Amorpheus.
  arXiv  Detail & Related papers  (2020-10-05T08:37:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.