Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention

• URL: http://arxiv.org/abs/2607.09889v1
• Date: Fri, 10 Jul 2026 18:28:21 GMT
• Title: Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention
• Abstract summary: We study a sparse cache that allocates a slot only when an input is novel, so its size tracks the number of distinct items rather than the number of tokens.<n>We develop it in two forms, a static cache with a fixed concentration and a surprise-adaptive variant whose concentration follows the recent novelty rate.
• Score: 0.48342038441006796
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fixed-state sequence models compress an unbounded past into a bounded state, which caps their associative recall at roughly the state dimension; attention escapes the cap by keeping a key-value entry for every token, at quadratic compute and a cache that grows with the sequence. We study the middle ground: a sparse cache that allocates a slot only when an input is novel, so its size tracks the number of distinct items rather than the number of tokens. The allocation rule is the DP-means clustering rule, the small-variance limit of a Dirichlet-process mixture, used not as latent-variable inference but as the key-value memory operator for a deep recurrent backbone. We develop it in two forms, a static cache with a fixed concentration and a surprise-adaptive variant whose concentration follows the recent novelty rate. On a controlled associative-recall benchmark with redundancy we show that the cache matches full-attention recall while storing only the distinct items, that it dominates a fixed-budget eviction cache on the recall-versus-size frontier, and that on a state-space backbone it answers both a recall query and a long-range aggregate at the lowest memory of any model tested. The allocation is learnable end to end: a two-parameter novelty-threshold gate trained on the task loss alone recovers the rule exactly, whereas an over-parameterized gate fails, so the operative ingredient is the inductive bias rather than capacity. The evidence is a family of controlled mechanism studies at modest scale, with the distinct-items property confirmed on four real streams (recommendation, systems logs, clinical events, and insurance claims); a real-backbone, real-corpus language validation is pursued in a companion study.

Related papers

• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• Raven: High-Recall Sequence Modeling with Sparse Memory Routing [88.47618436676748]
  We introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing.<n>Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade.
  arXiv  Detail & Related papers  (2026-07-28T07:04:32Z)
• Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams [0.48342038441006796]
  We develop that allocate-on-novelty cache as a working-memory component.<n>We show that novelty-gated attention reaches full-attention performance while attending to about half the tokens.<n>Experiments are small-scale and use only public data.
  arXiv  Detail & Related papers  (2026-07-11T18:58:22Z)
• What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents [1.0742675209112622]
  We show how four separate research communities have each learned to compact large language memory.<n>We argue that these are instances of one problem: a rate--distortion decision about what context-derived information to retain versus discard.<n>We make this lens precise with a single compaction objective and a layer-agnostic lower bound, use it to build a seven-axis taxonomy that classifies methods from across the stack uniformly.
  arXiv  Detail & Related papers  (2026-07-09T01:15:03Z)
• Interdomain Attention: Beyond Token-Level Key-Value Memory [15.708551010135407]
  We propose Interdomain Attention, which integrates an SSM into an attention module through kernel methods.<n>An attention kernel is approximated by a finite feature map, the resulting key features and values are projected onto a shared set of basis functions.<n>Each query attends to the compressed coefficients through its own feature map, recovering query-conditioned attention over a fixed-size state.
  arXiv  Detail & Related papers  (2026-05-23T01:18:00Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• A Parametric Memory Head for Continual Generative Retrieval [52.66674234249913]
  Generative information retrieval (GenIR) consolidates retrieval into a single neural model that decodes document identifiers (docids) directly from queries.<n>We show that sequential adaptation improves retrieval on newly added documents but substantially degrades performance on earlier slices.<n>We propose post-adaptation memory tuning (PAMT), a memory-only stabilization stage that augments an adapted model with a modular parametric memory head.
  arXiv  Detail & Related papers  (2026-04-25T17:38:51Z)
• Sparse Prefix Caching for Hybrid and Recurrent LLM Serving [0.7284556903703034]
  We formalize sparse prefix caching as checkpoint placement under a distribution over overlap depths.<n>For use cases where requests share a non-trivial prefix, we show that our method consistently improves the frontier traced by standard asymmetry on real-world data.<n>It preserves exact outputs, does not change the recurrent computation itself or require new recurrent update kernels.
  arXiv  Detail & Related papers  (2026-04-17T09:24:58Z)
• Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval [59.295767860331004]
  RF-Mem is a familiarity uncertainty-guided dual-path memory retriever.<n>It embeds human-like dual-process recognition into the retriever.<n>It consistently outperforms one-shot retrieval and full-context reasoning under fixed budget and latency constraints.
  arXiv  Detail & Related papers  (2026-03-10T06:31:44Z)
• Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View [16.8179962093575]
  We study memory in state-space language models using primacy and recency effects as behavioral tools to uncover how information is retained and forgotten over time.<n>Applying structured recall tasks to the Mamba architecture, we observe a consistent U-shaped accuracy profile, indicating strong performance at the beginning and end of input sequences.
  arXiv  Detail & Related papers  (2025-06-18T06:02:02Z)
• Efficient Inference of Vision Instruction-Following Models with Elastic Cache [76.44955111634545]
  We introduce Elastic Cache, a novel strategy for efficient deployment of instruction-following large vision-language models. We propose an importance-driven cache merging strategy to prune redundancy caches. For instruction encoding, we utilize the frequency to evaluate the importance of caches. Results on a range of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also notably outperforms existing pruning methods in language generation.
  arXiv  Detail & Related papers  (2024-07-25T15:29:05Z)
• Training-Free Exponential Context Extension via Cascading KV Cache [49.608367376911694]
  We introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens.<n>Our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens.
  arXiv  Detail & Related papers  (2024-06-24T03:59:17Z)
• Dual Cluster Contrastive learning for Person Re-Identification [78.42770787790532]
  We formulate a unified cluster contrastive framework, named Dual Cluster Contrastive learning (DCC) DCC maintains two types of memory banks: individual and centroid cluster memory banks. It can be easily applied for unsupervised or supervised person ReID.
  arXiv  Detail & Related papers  (2021-12-09T02:43:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.