Models Take Notes at Prefill: KV Cache Can Be Editable and Composable

• URL: http://arxiv.org/abs/2606.17107v1
• Date: Sun, 14 Jun 2026 15:31:23 GMT
• Title: Models Take Notes at Prefill: KV Cache Can Be Editable and Composable
• Abstract summary: Prefix caching reuses prefill only across an exactly shared prefix, so one changed field invalidates the entire downstream cache.<n>Yet overwriting the field's own key/value vectors and reusing the rest leaves the model acting on the old value.<n>The reason, established causally across four model families, is that at prefill the model has already written the field-conditioned conclusion onto downstream notes.<n>A salient erratum amends the notes; and with chain-of-thought, editing the field alone recovers the decision (1.00 at 8B, 1% compute), while without CoT it is ignored.
• Score: 0.3951796994513004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prefix caching reuses prefill only across an exactly shared prefix, so one changed field invalidates the entire downstream cache. Yet overwriting the field's own key/value vectors and reusing the rest leaves the model acting on the old value. The reason, established causally across four model families: at prefill the model has already written the field-conditioned conclusion onto downstream notes; the field's own key/value drives under 1% of the decision. Read as a notebook of memoized conclusions, two capabilities follow. (1) It is editable. A salient erratum amends the notes; and with chain-of-thought, editing the field alone recovers the decision (1.00 at 8B, ~1% compute), while without CoT it is ignored. (2) It is composable. The notes are position-portable, so a precompiled skill can be RoPE-repositioned and spliced into any context, indistinguishable from full recompute (logit cosine 0.90-0.999, twelve models) at O(L) rather than O(L^2) time-to-first-token. A unified edit+compose agent stays decision-identical to recompute at up to 14.9x lower latency. The approach applies to any per-token attention KV cache, validated across scale, quantization, Mixture-of-Experts, and multimodal caches, and extends to several attention variants through small adapters. Because the erratum is append-only, it composes with production prefix caching: in an online vLLM benchmark it keeps the prefix cache-aligned (98.5% hit-rate), cutting p90 time-to-first-token by 53-398x.

Related papers

• Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale [18.318746660954663]
  Block diffusion restores caching by decoding block-by-block.<n>We study this recipe at scale, pretraining three 3B block-diffusion denoisers on 300B tokens.
  arXiv  Detail & Related papers  (2026-09-09T23:37:47Z)
• HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation [56.14416807766864]
  Homomorphic encryption allows a server to run a language model directly on encrypted user prompts.<n>Existing approaches fix the iteration counts uniformly across the model rather than tailoring them to each site's error tolerance.<n>We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonline iteration counts learnable.
  arXiv  Detail & Related papers  (2026-09-01T18:02:02Z)
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention [0.48342038441006796]
  We study a sparse cache that allocates a slot only when an input is novel, so its size tracks the number of distinct items rather than the number of tokens.<n>We develop it in two forms, a static cache with a fixed concentration and a surprise-adaptive variant whose concentration follows the recent novelty rate.
  arXiv  Detail & Related papers  (2026-07-10T18:28:21Z)
• Leyline: KV Cache Directives for Agentic Inference [0.11099872871193028]
  Leyline is a serving-side primitive for agentic editing.<n>A declarative directive 4-tuple separates what to edit from how to preserve position correctness.<n>A ten-line truncation rule routed through the same interface lifts agentic solve rate by +14.3 pp on debug-gym.
  arXiv  Detail & Related papers  (2026-05-31T07:13:15Z)
• Nectar: Neural Estimation of Cached-Token Attention via Regression [36.709992864731866]
  We show how a compact neural network can be used to evaluate softmax attention over a fixed long context.<n>Nectar fits two networks per layer and KV-head: a target network that predicts the attention output and a score network that predicts the log-normalizer.<n>We report experiments on models from 1.7B to 8B parameters across five long-context datasets.
  arXiv  Detail & Related papers  (2026-05-10T21:51:36Z)
• HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing [12.303016850029683]
  Large language models encode vast factual knowledge that inevitably becomes outdated or incorrect after deployment.<n>One line of work installs new facts by directly modifying base weights through locate-then-edit procedures.<n>A complementary line leaves base weights intact and routes edits through external memory, but it faces routing challenges.<n>We propose HoReN, a codebook-based editor with enhanced routing built on three ideas.
  arXiv  Detail & Related papers  (2026-05-02T15:51:31Z)
• The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference [3.378773775514883]
  Key-value (KV) cache is widely treated as essential state in transformer inference.<n>We prove that keys and values at every layer are deterministic projections of the residual stream.<n>We build on this result with KV-Direct, a bounded-memory inference scheme.
  arXiv  Detail & Related papers  (2026-03-20T05:59:50Z)
• Deliberation in Latent Space via Differentiable Cache Augmentation [48.228222586655484]
  We show that a frozen large language model can be augmented with an offline coprocessor that operates on the model's key-value (kv) cache.<n>This coprocessor augments the cache with a set of latent embeddings designed to improve the fidelity of subsequent decoding.<n>We show experimentally that when a cache is augmented, the decoder achieves lower perplexity on numerous subsequent tokens.
  arXiv  Detail & Related papers  (2024-12-23T18:02:25Z)
• Training-Free Exponential Context Extension via Cascading KV Cache [49.608367376911694]
  We introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens.<n>Our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens.
  arXiv  Detail & Related papers  (2024-06-24T03:59:17Z)
• Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching [56.286064975443026]
  We make an interesting and somehow surprising observation: the computation of a large proportion of layers in the diffusion transformer, through a caching mechanism, can be readily removed even without updating the model parameters. We introduce a novel scheme, named Learningto-Cache (L2C), that learns to conduct caching in a dynamic manner for diffusion transformers. Experimental results show that L2C largely outperforms samplers such as DDIM and DPM-r, alongside prior cache-based methods at the same inference speed.
  arXiv  Detail & Related papers  (2024-06-03T18:49:57Z)
• You Only Cache Once: Decoder-Decoder Architectures for Language Models [132.4064488592704]
  We introduce a decoder-decoder architecture, YOCO, for large language models. YOCO only caches key-value pairs once. The overall model behaves like a decoder-only Transformer, although YOCO only caches once.
  arXiv  Detail & Related papers  (2024-05-08T17:57:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.