NGM: A Plug-and-Play Training-Free Memory Module for LLMs

• URL: http://arxiv.org/abs/2605.16893v1
• Date: Sat, 16 May 2026 09:12:52 GMT
• Title: NGM: A Plug-and-Play Training-Free Memory Module for LLMs
• Abstract summary: N-gram Memory (NGM) is a training-free, plug-and-play module composed of a Causal N-Gram and a Cosine-Gated Memory.<n>NGM eliminates the need to train separate N-gram embeddings from scratch.<n>We evaluate NGM on the Qwen3 series from 0.6B to 14B across eight benchmarks.
• Score: 25.667968001687342
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent studies introduce conditional memory modules that decouple knowledge storage from neural computation, enabling more direct knowledge access. Compared to MoE, which relies on dynamic computation paths, explicit lookup provides a more efficient knowledge retrieval mechanism. However, these approaches still depend on learned memory embeddings, requiring additional training and limiting flexibility. To address this, we propose N-gram Memory (NGM), a training-free, plug-and-play module composed of a Causal N-Gram Encoder and a Cosine-Gated Memory Injector. The Causal N-Gram Encoder directly averages the pretrained token embeddings of the backbone model to construct N-gram representations, thereby eliminating the need to train separate N-gram embeddings from scratch. This design requires neither an additional memory table nor a retrieval pipeline. The Cosine-Gated Memory Injector then uses a non-parametric cosine gate with ReLU to modulate the retrieved embeddings into the contextual representations. We evaluate NGM on the Qwen3 series from 0.6B to 14B across eight benchmarks. NGM improves average performance by 0.5 to 1.2 points, with particularly clear gains on code generation and knowledge-intensive tasks (e.g., +3.0 on LiveCodeBench and +3.03 on GPQA for Qwen3-14B). Moreover, NGM also improves performance in multimodal benchmarks (e.g., MMStar +1.53 on Qwen3-VL-2B).

Related papers

• Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner [55.80248281761638]
  multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning.<n>We propose DG-Mem, a dual-grained agentic memory framework that augments a frozen MLLM with a non- externally stored memory built once from training-time rollouts and consulted read-only at test time.
  arXiv  Detail & Related papers  (2026-08-24T13:56:01Z)
• Rank-Order N-of-M Codes for Sparse Distributed Memory: Disentangling Representation and Learning Effects in Noise Robustness Against Contemporary Neuromorphic Architectures [0.0]
  This paper evaluates rank-order N-of-M encoding as an alternative to the threshold-binary encoder in CALM.<n> experiments on GloVe-100 embeddings confirm this small but consistent encoding benefit on real structured data.<n>Results identify which components of the original rank-order SDM architecture provide measurable benefits for contemporary memory-augmented AI systems.
  arXiv  Detail & Related papers  (2026-07-03T05:21:38Z)
• Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory [65.39827296429527]
  Scaling conditional memory offers a promising way to increase language-model capacity.<n>Existing methods such as Engram learn large memory tables from scratch during pre-training.<n>We propose Memory Grafting, a conditional memory scaling method that utilizes frozen hidden states from a grafting model as conditional n-gram memory.
  arXiv  Detail & Related papers  (2026-05-20T09:35:04Z)
• Beyond GEMM-Centric NPUs: Enabling Efficient Diffusion LLM Sampling [14.471123653746275]
  Diffusion Large Language Models (dLLMs) introduce iterative denoising to enable parallel token generation.<n>Our design employs lightweight non-GEMM vector primitives, in-place memory reuse strategies, and a decoupled mixed-precision memory hierarchy.
  arXiv  Detail & Related papers  (2026-01-28T15:37:50Z)
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models [42.816060150754645]
  We introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic $N$-gram embedding for O(1) lookup.<n>We scale Engram to 27B parameters, achieving superior performance over a strictly iso- parameter and iso-FLOPs MoE baseline.<n>We envision conditional memory as an indispensable modeling primitive for next-generation sparse models.
  arXiv  Detail & Related papers  (2026-01-12T09:54:49Z)
• CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs [14.5426213901124]
  We present CodeGEMM, a codebook-centric GEMM kernel that replaces dequantization with precomputed inner products between centroids and activations stored in a lightweight Psumbook.<n>On Llama-3 models, CodeGEMM delivers 1.83x (8B) and 8.93x (70B) speedups in the 2-bit configuration compared to state-of-the-art codebook-based quantization at comparable accuracy.
  arXiv  Detail & Related papers  (2025-12-19T06:16:32Z)
• Generative Binary Memory: Pseudo-Replay Class-Incremental Learning on Binarized Embeddings [1.3980986259786223]
  This paper introduces Generative Binary Memory (GBM), a novel CIL pseudo-replay approach which generates synthetic binary pseudo-exemplars.<n>With a specifically-designed feature binarizer, GBM effectively models the multi-modal characteristics of class distributions, in a latent, binary space.
  arXiv  Detail & Related papers  (2025-03-13T13:11:59Z)
• A Universal Framework for Compressing Embeddings in CTR Prediction [68.27582084015044]
  We introduce a Model-agnostic Embedding Compression (MEC) framework that compresses embedding tables by quantizing pre-trained embeddings.<n>Our approach consists of two stages: first, we apply popularity-weighted regularization to balance code distribution between high- and low-frequency features.<n> Experiments on three datasets reveal that our method reduces memory usage by over 50x while maintaining or improving recommendation performance.
  arXiv  Detail & Related papers  (2025-02-21T10:12:34Z)
• LM2: Large Memory Models [11.320069795732058]
  This paper introduces the Large Memory Model (LM2), a decoder-only Transformer architecture enhanced with an auxiliary memory module.<n> Experimental results on the BABILong benchmark demonstrate that the LM2model outperforms both the memory-augmented RMT model by 37.1% and the baseline Llama-3.2 model by 86.3% on average across tasks.
  arXiv  Detail & Related papers  (2025-02-09T22:11:42Z)
• Memory Layers at Scale [67.00854080570979]
  This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale.<n>On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the budget, as well as mixture-of-expert models when matched for both compute and parameters.<n>We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters.
  arXiv  Detail & Related papers  (2024-12-12T23:56:57Z)
• Compacting Binary Neural Networks by Sparse Kernel Selection [58.84313343190488]
  This paper is motivated by a previously revealed phenomenon that the binary kernels in successful BNNs are nearly power-law distributed. We develop the Permutation Straight-Through Estimator (PSTE) that is able to not only optimize the selection process end-to-end but also maintain the non-repetitive occupancy of selected codewords. Experiments verify that our method reduces both the model size and bit-wise computational costs, and achieves accuracy improvements compared with state-of-the-art BNNs under comparable budgets.
  arXiv  Detail & Related papers  (2023-03-25T13:53:02Z)
• Neural Attentive Circuits [93.95502541529115]
  We introduce a general purpose, yet modular neural architecture called Neural Attentive Circuits (NACs) NACs learn the parameterization and a sparse connectivity of neural modules without using domain knowledge. NACs achieve an 8x speedup at inference time while losing less than 3% performance.
  arXiv  Detail & Related papers  (2022-10-14T18:00:07Z)
• GLEAM: Greedy Learning for Large-Scale Accelerated MRI Reconstruction [50.248694764703714]
  Unrolled neural networks have recently achieved state-of-the-art accelerated MRI reconstruction. These networks unroll iterative optimization algorithms by alternating between physics-based consistency and neural-network based regularization. We propose Greedy LEarning for Accelerated MRI reconstruction, an efficient training strategy for high-dimensional imaging settings.
  arXiv  Detail & Related papers  (2022-07-18T06:01:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.