AgentKV: Phase-Aware KV Eviction for Agentic LLMs

• URL: http://arxiv.org/abs/2609.14872v1
• Date: Mon, 14 Sep 2026 00:42:53 GMT
• Title: AgentKV: Phase-Aware KV Eviction for Agentic LLMs
• Abstract summary: KV-eviction methods score cached keys against representative queries drawn from the most recent tokens.<n>We show that future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces.<n>We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union.
• Score: 15.992100550192765
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV-eviction methods score cached keys against representative queries drawn from the most recent tokens, assuming future attention resembles recent attention. We show that agentic generation violates this assumption: future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces, so recency representatives systematically undervalue keys that upcoming phases will need. We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union. We further implement AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online. Across two models, six task domains, and three KV budgets each, AGENTKV improves task score by 5.5 points on average over R-KV and 5.3 over Tri-attention. Relative to upstream full-KV SGLang, AGENTKV improves output-token throughput by up to 1.80x. Code: https://github.com/LiuTaowen-Tony/agentkv.

Related papers

• BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference [20.58039191111583]
  Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation.<n>The resulting key-value ( KV) cache grows linearly with sequence length and creates severe memory bottlenecks.<n>We propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster.
  arXiv  Detail & Related papers  (2026-09-04T10:23:16Z)
• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration [67.17709877327232]
  Cate KV is a hybrid KV cache method that retains only critical token information for consistent heads.<n>Cate KV reduces memory usage by up to $2.72times$ and accelerates decoding by $2.18times$ in single-sample inputs.
  arXiv  Detail & Related papers  (2026-08-31T06:02:37Z)
• Information-Aware KV Cache Compression for Long Reasoning [71.77807044404499]
  Existing KV cache compression methods mainly rely on attention weights to estimate token importance.<n>We propose textbfInfoKV, an entropy-aware KV cache compression framework that incorporates information-theoretic signals.
  arXiv  Detail & Related papers  (2026-06-25T11:03:37Z)
• IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference [12.04455190856202]
  Multi-turn LLM agents fan short queries into long trajectories of tool calls, search results, and intermediate reasoning.<n>We introduce IntentKV, learned KV pruning that keeps the base LLM frozen.<n>IntentKV matches the no-pruning full-cache baseline with almost no accuracy drop under tight KV budgets.
  arXiv  Detail & Related papers  (2026-06-06T15:54:48Z)
• TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks [20.031258206130154]
  We introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme for agentic workloads.<n>We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management.
  arXiv  Detail & Related papers  (2026-05-16T21:58:28Z)
• TriAttention: Efficient Long Reasoning with Trigonometric KV Compression [42.4775416331056]
  Extended reasoning in large language models (LLMs) creates severe KV cache memory bottlenecks.<n>Leading KV cache compression methods estimate KV importance using attention scores from recent post-RoPE queries.<n>We propose TriAttention to estimate key importance by leveraging these centers.
  arXiv  Detail & Related papers  (2026-04-06T17:58:42Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• G-KV: Decoding-Time KV Cache Eviction with Global Attention [57.47409249054187]
  Large language models (LLMs) excel in complex tasks but encounter significant computational and memory challenges due to long sequence lengths.<n> KV cache compression has emerged as an effective approach to greatly enhance the efficiency of reasoning.<n>We propose G-KV, a KV cache eviction method that employs a global scoring mechanism, combining local and historical attention scores to more accurately assess token importance.
  arXiv  Detail & Related papers  (2025-11-29T14:21:33Z)
• Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution [2.894551569099569]
  We introduce $textbfExpected Attention, a training-free compression method that estimates KV pairs importance by predicting how future queries will attend to them.<n>Our method operates seamlessly across both prefilling and decoding phases, consistently outperforming state-of-the-art baselines in both scenarios.<n>$textbfwe release KVPress, a comprehensive library to enable researchers to implement and benchmark KV cache compression methods.
  arXiv  Detail & Related papers  (2025-10-01T08:12:14Z)
• Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction [53.83828564664595]
  Large language models (LLMs) utilize key-value ( KV) cache to store historical information during sequence processing.<n>Current methods for KV cache eviction typically utilize the last window from the pre-filling phase as queries to compute the KV importance scores for eviction.<n>We propose Judge Q, a novel training method which incorporates a soft token list.
  arXiv  Detail & Related papers  (2025-09-13T03:34:12Z)
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction [37.97434606840326]
  Transformer-based large language models (LLMs) cache context as key-value ( KV) pairs during inference.<n>As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency.<n>This paper introduces KVzip, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries.
  arXiv  Detail & Related papers  (2025-05-29T13:05:47Z)
• More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression [71.42818367729573]
  In large language models (LLMs), the memory usage of KV cache has become a critical bottleneck during inference.<n>The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately.<n>In this paper, we comprehensively investigate the token-precision trade-off in KV cache compression.
  arXiv  Detail & Related papers  (2024-12-17T09:20:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.