IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference

• URL: http://arxiv.org/abs/2606.09916v1
• Date: Sat, 06 Jun 2026 15:54:48 GMT
• Title: IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference
• Abstract summary: Multi-turn LLM agents fan short queries into long trajectories of tool calls, search results, and intermediate reasoning.<n>We introduce IntentKV, learned KV pruning that keeps the base LLM frozen.<n>IntentKV matches the no-pruning full-cache baseline with almost no accuracy drop under tight KV budgets.
• Score: 12.04455190856202
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-turn LLM agents fan short queries into long trajectories of tool calls, search results, and intermediate reasoning. Both KV memory and KV read bandwidth grow by orders of magnitude across a single trajectory, making the key-value (KV) cache, not parameter compute, the dominant serving bottleneck for long-horizon agents. We introduce IntentKV, learned KV pruning that keeps the base LLM frozen. IntentKV maintains a session-level QueryMemory of cross-turn intent, scores live history tokens with a memory-attention rule, and adds a zero-initialized residual head with cross-attention over current-query K-vectors. To stay composable with prefix caches, eviction is a slot-map redirection: dropped positions route to a sentinel dead slot while surviving K/V rows, RoPE phases, and slot identities stay in place. IntentKV matches the no-pruning full-cache baseline with almost no accuracy drop under tight KV budgets: at an 8k KV budget, mean peak request tokens drop 23.9% on Qwen3-8B and 30.7% on Qwen2.5-14B. On the 100 longest BCP queries that all methods complete on Qwen2.5-14B, IntentKV-8k further cuts worst-case peak request tokens from 92.3k to 20.5k, a 77.8% reduction, and worst-case raw KV reads from 411M to 31M, a 92.6% reduction.

Related papers

• AgentKV: Phase-Aware KV Eviction for Agentic LLMs [15.992100550192765]
  KV-eviction methods score cached keys against representative queries drawn from the most recent tokens.<n>We show that future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces.<n>We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union.
  arXiv  Detail & Related papers  (2026-09-14T00:42:53Z)
• KVpop -- Key-Value Cache Compression with Predictive Online Pruning [22.664867330302517]
  We introduce KVpop, which learns a fixed-budget KV eviction policy by directly supervising the keep-or-drop decision.<n>The scorer is trained against a novel future-attention target, computed efficiently without materializing dense attention maps.<n> KVpop retains 98% of full-attention performance on Qwen3-4B at 75% KV cache compression and 97% at 88% compression, consistently outperforming established eviction baselines.
  arXiv  Detail & Related papers  (2026-07-06T13:32:34Z)
• CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM [8.82843279804122]
  We introduce CONF-KV, a KV-cache manager that chooses the per-step cache budget.<n> tokens are ranked by a composite of accumulated attention mass and recency.<n>On 75 VisualWebArena tasks it retains 95.3% of full- KV success at 2.8 times lower peak memory.
  arXiv  Detail & Related papers  (2026-05-24T00:15:34Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation [22.835149054167122]
  We propose Prophet KV, a user-query-driven KV Cache reuse method for RAG scenarios.<n> Prophet KV prioritizes tokens based on their semantic relevance to the user query.<n>Our evaluation results show that Prophet KV retains 96%-101% of full-prefill accuracy with only a 20% recomputation ratio.
  arXiv  Detail & Related papers  (2026-01-31T09:53:31Z)
• Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs? [79.58770714228983]
  Language models handle increasingly long contexts for tasks such as book summarization.<n>This leads to growing memory costs for the key-value ( KV) cache.<n>Many prior works have proposed ways of discarding KVs from memory, but their approaches are tailored to favorable settings.<n>We propose the * KV footprint* as a unified metric, which accounts for both the amount of KV entries stored and their lifespan in memory.
  arXiv  Detail & Related papers  (2025-06-20T16:21:12Z)
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models [77.84539432982307]
  We propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV)<n>R-KV preserves nearly 100% of the full KV cache performance using only 10% of the KV cache.<n>Remarkably, R-KV even achieves 105% of full KV cache performance with 16% of the KV cache.
  arXiv  Detail & Related papers  (2025-05-30T02:03:24Z)
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction [37.97434606840326]
  Transformer-based large language models (LLMs) cache context as key-value ( KV) pairs during inference.<n>As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency.<n>This paper introduces KVzip, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries.
  arXiv  Detail & Related papers  (2025-05-29T13:05:47Z)
• KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse [17.301289617498448]
  We present a KV cache management module that shares the KV cache across requests under multi-tenant scenarios.<n> KVShare reduces TTFT by up to 9.39x and increases 1.2x of the throughput compared to the full KV recompute.<n> KVShare achieves 20.38% boost in terms of accuracy compared to SOTA methods.
  arXiv  Detail & Related papers  (2025-03-17T16:43:35Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression [71.42818367729573]
  In large language models (LLMs), the memory usage of KV cache has become a critical bottleneck during inference.<n>The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately.<n>In this paper, we comprehensively investigate the token-precision trade-off in KV cache compression.
  arXiv  Detail & Related papers  (2024-12-17T09:20:31Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.