Probing the Limits of Compressive Memory: A Study of Infini-Attention in Small-Scale Pretraining

• URL: http://arxiv.org/abs/2512.23862v1
• Date: Mon, 29 Dec 2025 21:02:14 GMT
• Title: Probing the Limits of Compressive Memory: A Study of Infini-Attention in Small-Scale Pretraining
• Abstract summary: This study investigates small-scale pretraining for Small Language Models (SLMs)<n>Infini-attention builds a compressed memory from past segments while preserving local attention.<n>Our findings suggest that achieving robust long-context capability in SLMs benefits from architectural memory like Infini-attention.
• Score: 4.504354174605603
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This study investigates small-scale pretraining for Small Language Models (SLMs) to enable efficient use of limited data and compute, improve accessibility in low-resource settings and reduce costs. To enhance long-context extrapolation in compact models, we focus on Infini-attention, which builds a compressed memory from past segments while preserving local attention. In our work, we conduct an empirical study using 300M-parameter LLaMA models pretrained with Infini-attention. The model demonstrates training stability and outperforms the baseline in long-context retrieval. We identify the balance factor as a key part of the model performance, and we found that retrieval accuracy drops with repeated memory compressions over long sequences. Even so, Infini-attention still effectively compensates for the SLM's limited parameters. Particularly, despite performance degradation at a 16,384-token context, the Infini-attention model achieves up to 31% higher accuracy than the baseline. Our findings suggest that achieving robust long-context capability in SLMs benefits from architectural memory like Infini-attention.

Related papers

• Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling [65.65882819517455]
  We propose a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss.<n> Experimental results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths.
  arXiv  Detail & Related papers  (2026-07-03T05:39:00Z)
• Surface-Form Neural Sparse Retrieval: Robust Fuzzy Matching for Industrial Music Search [6.820455959856271]
  Music search at the scale of Amazon Music presents a unique challenge: queries frequently deviate from indexed metadata due to misspellings, transpositions, and phonetic variations.<n>Our existing learning-to-retrieve system, the High Confidence Index (HCI), learns query-entity associations from customer behavior.<n>Traditional n-gram matching enables this exploration but suffers from poor semantic robustness and high noise.<n>We present a textbfrobust neural sparse retrieval system designed to maximize exploration efficiency.
  arXiv  Detail & Related papers  (2026-05-18T02:29:25Z)
• AllMem: A Memory-centric Recipe for Efficient Long-context Modeling [32.025154452526856]
  Large Language Models (LLMs) encounter significant performance bottlenecks in long-sequence tasks.<n>We introduce textscAllMem, a novel and efficient hybrid architecture that integrates Sliding Window Attention (SWA) with non-linear Test-Time Training (TTT) memory networks.
  arXiv  Detail & Related papers  (2026-02-14T09:04:28Z)
• Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoning [62.680551162054975]
  We introduce an end-to-end framework where LLMs learn to self-regulate the granularity of the reasoning steps through dynamic summarization.<n>We apply reinforcement learning to incentivize this capability further, uncovering a critical insight: the accuracy gap between the highly efficient Fold mode and the exhaustive Unfold mode progressively narrows.<n>Our Accordion-Thinker demonstrates that with learned self-compression, LLMs can tackle complex reasoning tasks with minimal dependency token overhead.
  arXiv  Detail & Related papers  (2026-02-03T08:34:20Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing [19.577278316436807]
  Large Language Models (LLMs) are limited by the context window size.<n>We propose a novel method that leverages the LLMs's own attention information to enable accurate retrieval.<n>InfiniRetri achieves 100% accuracy in the Needle-In-a-Haystack(NIH) test over 1M tokens using a 0.5B parameter model.
  arXiv  Detail & Related papers  (2025-02-18T15:45:36Z)
• InfiniPot: Infinite Context Processing on Memory-Constrained LLMs [17.111422610001227]
  InfiniPot is a novel KV cache control framework designed to enable pre-trained Large Language Models to manage extensive sequences efficiently. InfiniPot effectively maintains critical data even without access to future context. This work represents a substantial advancement toward making Large Language Models applicable to a broader range of real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-02T13:09:41Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• ReAttention: Training-Free Infinite Context with Finite Attention Scope [65.91272939057592]
  ReAttention is a training-free approach to support infinite context with finite attention scope under sufficient memory resources.<n>We validate the performance of ReAttention on the LongBench, L-Eval, and InfiniteBench and demonstrate that it is on par with traditional methods.<n>We also apply ReAttention on mainstream LLMs, including LLaMA3.1-8B and Mistral-v0.3-7B, enabling them to support context lengths of at least 1M and even expanding the context length of LLaMA3.2-3B-chat by 128$times$ to 4M without any further training in Needle-In-
  arXiv  Detail & Related papers  (2024-07-21T14:23:37Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention [6.713196608291278]
  This work introduces an efficient method to scale Transformer-based Large Language Models to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention.
  arXiv  Detail & Related papers  (2024-04-10T16:18:42Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)
• Online Continual Learning Without the Storage Constraint [67.66235695269839]
  We contribute a simple algorithm, which updates a kNN classifier continually along with a fixed, pretrained feature extractor. It can adapt to rapidly changing streams, has zero stability gap, operates within tiny computational budgets, has low storage requirements by only storing features. It can outperform existing methods by over 20% in accuracy on two large-scale online continual learning datasets.
  arXiv  Detail & Related papers  (2023-05-16T08:03:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.