CachePrune: Privacy-Aware and Fine-Grained KV Cache Sharing for Efficient LLM Inference

• URL: http://arxiv.org/abs/2605.23640v1
• Date: Fri, 22 May 2026 13:54:21 GMT
• Title: CachePrune: Privacy-Aware and Fine-Grained KV Cache Sharing for Efficient LLM Inference
• Abstract summary: CachePrune is a privacy-aware KV cache sharing mechanism that enables fine-grained reuse of KV entries across requests.<n>We show that it eliminates direct leakage through KV cache reuse side channels while reducing TTFT by 4.5x and increasing cache hit rates by 44%.
• Score: 21.373440644465607
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) rely on Key-Value (KV) caching to accelerate inference, and many serving systems further share the KV cache across users' requests to reduce redundant computation. While widely adopted, unrestricted cross-user sharing introduces side-channel vulnerabilities, allowing an adversary to infer user inputs by probing for cache reuse. Existing defenses disable sharing entirely to prevent leakage; yet such a coarse-grained strategy sacrifices substantial reuse potential, since prompts often include large portions of privacy-irrelevant segments, such as system instructions or publicly accessible materials. Building on this, we present CachePrune, a privacy-aware KV cache sharing mechanism that enables fine-grained reuse of KV entries across requests. Realizing such fine granularity requires token-level cache management, as reusable segments vary in length and position due to sensitivity masking, making reuse more complex than the fixed-size or sentence-level chunking used in existing coarse-grained schemes. Specifically, CachePrune makes fine-grained reuse practical by addressing two key challenges: accurately and efficiently deriving reusable KV segments and efficiently retrieving them over variable-length spans. We implement CachePrune on top of vLLM and evaluate it on three datasets, showing that it eliminates direct leakage through KV cache reuse side channels while reducing TTFT by 4.5x and increasing cache hit rates by 44% compared with state-of-the-art approaches.

Related papers

• VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference [17.932004205808486]
  VisCache is a framework for coarse-to-fine textbfVisual KV textbfCache pruning without training.<n>PruneKV is a surgical KV compression algorithm tailored to the attention dynamics of Vision Large Language Models.<n>VisCache consistently outperforms existing baselines.
  arXiv  Detail & Related papers  (2026-08-25T04:52:05Z)
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• Recency/Frequency Adaptive KV Caching for Large Language Model Serving [7.261125311665803]
  Key-value ( KV) caching is a powerful technique for accelerating large language model inference and generation.<n>Inference workloads are large and diverse, which makes them difficult to cache effectively.<n>We integrate adaptive caching that dynamically allocates cache space between recently and frequently occurring KV blocks.
  arXiv  Detail & Related papers  (2026-06-19T09:05:01Z)
• MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering [75.0394545769057]
  KV-caching stores the Key-Value of the historical tokens via LLM prefill.<n>MuKV is a method that features a multi-grained KV cache compression module and a semi-hierarchical retrieval approach.<n> Experiments on long-streaming VideoQA benchmarks show that MuKV significantly improves answer accuracy, without sacrificing memory and online QA efficiency.
  arXiv  Detail & Related papers  (2026-05-21T10:13:03Z)
• Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction [65.710271475739]
  We introduce a global retention-based KV eviction method that learns each token's future utility under a unified memory budget.<n>We show that our method substantially reduces KV memory while matching or surpassing full-cache inference.<n>These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.
  arXiv  Detail & Related papers  (2026-05-10T16:47:50Z)
• KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression [3.9838043625801967]
  Reasoning language models generate long chain-of-thought (CoT)<n>CoT accumulates a massive KV cache during the decoding phase.<n> KV cache compression has emerged as a promising technique for reducing memory overhead.
  arXiv  Detail & Related papers  (2026-05-01T15:54:13Z)
• SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching [0.8307668828380427]
  We propose textitSemShareKV, a KV cache sharing and compression framework for large language models (LLMs)<n>Instead of relying on exact token matches, SemShareKV applies fuzzy token matching using locality-sensitive hashing (LSH) on token embeddings and incorporates Rotary Position Embedding (RoPE) to better preserve positional information.<n> Experiments on diverse summarization datasets show up to 6.25$times$ speedup and 42% lower GPU memory usage with 5k tokens input, with negligible quality degradation.
  arXiv  Detail & Related papers  (2025-09-29T14:16:13Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• SCBench: A KV Cache-Centric Analysis of Long-Context Methods [61.025422435235456]
  We introduce SCBench, a benchmark for evaluating long-context methods from a KV cachecentric perspective.<n>We provide an extensive KV cache-centric analysis of eight categories long-context solutions, including Gated Linear RNNs and Mamba-Attention hybrids.<n>Our findings show that sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding with O(n) memory and sub-O(n2) pre-filling perform robustly.
  arXiv  Detail & Related papers  (2024-12-13T17:59:52Z)
• CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion [15.344568214955688]
  Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts.<n>To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input.<n>This paper tackles just one challenge: how to quickly combine their precomputed KV caches in order to achieve the same generation quality as the expensive full prefill.
  arXiv  Detail & Related papers  (2024-05-26T06:00:17Z)
• Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention [13.041210267981613]
  CachedAttention is a new attention mechanism that enables reuse of KV caches across multi-turn conversations. It significantly decreases the time to the first token (TTFT) by up to 87%, improves the prompt prefilling throughput by up to 7.8$times$ for multi-turn conversations, and reduces the end-to-end inference cost by up to 70%.
  arXiv  Detail & Related papers  (2024-03-23T10:42:49Z)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference [78.65321721142624]
  We focus on a memory bottleneck imposed by the key-value ( KV) cache. Existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs. We propose LESS, a simple integration of a constant sized cache with eviction-based cache methods.
  arXiv  Detail & Related papers  (2024-02-14T18:54:56Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.