Forget, Then Recall: Learnable Compression and Selective Unfolding via Gist Sparse Attention

• URL: http://arxiv.org/abs/2604.20920v1
• Date: Wed, 22 Apr 2026 04:22:32 GMT
• Title: Forget, Then Recall: Learnable Compression and Selective Unfolding via Gist Sparse Attention
• Abstract summary: Scaling large language models to long contexts is challenging due to the quadratic computational cost of full attention.<n>We introduce selective unfolding via GSA, which first compresses the context into gist tokens, then selects the most relevant gists.<n>This yields a simple coarse-to-fine mechanism that combines compact global representations with targeted access to fine-leaved evidence.
• Score: 7.694795562178033
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling large language models to long contexts is challenging due to the quadratic computational cost of full attention. Mitigation approaches include KV-cache selection or compression techniques. We instead provide an effective and end-to-end learnable bridge between the two without requiring architecture modification. In particular, our key insight is that interleaved gist compression tokens -- which provide a learnable summary of sets of raw tokens -- can serve as routing signals for sparse attention. Building on this, we introduce selective unfolding via GSA, which first compresses the context into gist tokens, then selects the most relevant gists, and subsequently restores the corresponding raw chunks for detailed attention. This yields a simple coarse-to-fine mechanism that combines compact global representations with targeted access to fine-grained evidence. We further incorporate this process directly into training in an end-to-end fashion, avoiding the need for external retrieval modules. In addition, we extend the framework hierarchically via recursive gist-of-gist construction, enabling multi-resolution context access with logarithmic per-step decoding complexity. Empirical results on LongBench and RAG benchmarks demonstrate that our method consistently outperforms other compression baselines as well as inference-time sparse attention methods across compression ratios from $8\times$ to $32\times$. The code is available at: https://github.com/yuzhenmao/gist-sparse-attention/

Related papers

• From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors [25.32575172623139]
  Long-context large language models remain computationally expensive to run and often fail to reliably process very long inputs.<n>We propose a training-free and model-agnostic compression framework that selects a compact set of sentences guided by structural graph priors.<n>Our method constructs a sparse hybrid sentence graph that combines mutual k-NN semantic edges with short-range sequential edges.
  arXiv  Detail & Related papers  (2026-04-25T12:40:54Z)
• Multi-Vector Index Compression in Any Modality [73.7330345057813]
  Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos.<n>We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC)<n>AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation.
  arXiv  Detail & Related papers  (2026-02-24T18:57:33Z)
• Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning [47.87361916374891]
  We propose a framework for efficient long-context inference based on chunk-wise compression and selective memory recall.<n>The framework segments long inputs into chunks and encodes each chunk into compressed memory representations using a learned compressor.<n>It achieves up to a 2 times reduction in peak GPU memory usage and a 6 times inference speedup over MemAgent.
  arXiv  Detail & Related papers  (2026-02-09T08:33:11Z)
• CompLLM: Compression for Long Context Q&A [47.90063873976842]
  We introduce CompLLM, a soft compression technique designed for practical deployment.<n>Instead of processing the context holistically, CompLLM divides it into segments and compresses each one independently.<n>Our experiments show that with a 2x compression rate, at high context lengths CompLLM speeds up Time To First Token (TTFT) by up to 4x and reduces the KV cache size by 50%.
  arXiv  Detail & Related papers  (2025-09-23T16:49:43Z)
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression [86.33995240043936]
  UniGist is a sequence-level long-context compression framework for large language models.<n>It efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner.<n>Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings.
  arXiv  Detail & Related papers  (2025-09-19T08:47:37Z)
• Sentinel: Attention Probing of Proxy Models for LLM Context Compression with an Understanding Perspective [29.50363211934763]
  Retrieval-augmented generation (RAG) enhances large language models with external context, but retrieved passages are often lengthy, noisy, or exceed input limits.<n>We propose Sentinel, a lightweight sentence-level compression framework that reframes context filtering as an attention-based understanding task.
  arXiv  Detail & Related papers  (2025-05-29T09:24:12Z)
• R1-Compress: Long Chain-of-Thought Compression via Chunk Compression and Search [61.4807238517108]
  Chain-of-Thought (CoT) reasoning enhances large language models (LLMs) by enabling step-by-step problem-solving.<n>CoT's extension to Long-CoT introduces substantial computational overhead due to increased token length.<n>We propose R1-Compress, a two-stage chunk-level compression framework that preserves both local information and coherence.
  arXiv  Detail & Related papers  (2025-05-22T16:06:59Z)
• Concise and Precise Context Compression for Tool-Using Language Models [60.606281074373136]
  We propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. Results on API-Bank and APIBench show that our approach reaches a performance comparable to the upper-bound baseline under up to 16x compression ratio.
  arXiv  Detail & Related papers  (2024-07-02T08:17:00Z)
• Context Compression for Auto-regressive Transformers with Sentinel Tokens [37.07722536907739]
  We propose a plug-and-play approach that is able to incrementally compress the intermediate activation of a specified span of tokens into compact ones. Experiments on both in-domain language modeling and zero-shot open-ended document generation demonstrate the advantage of our approach.
  arXiv  Detail & Related papers  (2023-10-12T09:18:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.