C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

• URL: http://arxiv.org/abs/2607.17715v1
• Date: Mon, 20 Jul 2026 09:09:23 GMT
• Title: C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
• Abstract summary: Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning.<n>To mitigate the growing inference cost, recent work has explored key-value ( KV) cache reuse to reduce redundant prefill computation.<n>Existing reuse methods primarily focus on savings and overlook a critical bottleneck in long-context serving: the cost of storing and accessing KV caches.
• Score: 31.428047145437773
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning. To mitigate the growing inference cost, recent work has explored key-value (KV) cache reuse to reduce redundant prefill computation. However, existing reuse methods primarily focus on computation savings and overlook a critical bottleneck in long-context LLM serving: the cost of storing and accessing large KV caches. While KV compression appears to be a natural complement, naively combining compression with non-prefix KV reuse often leads to severe accuracy degradation. In this work, we propose C$^2$KV, a unified framework for non-prefix KV reuse that jointly optimizes KV extraction and inference-time concatenation. C$^2$KV learns a composable and compressed KV cache manifold that is explicitly designed to be position-agnostic. Our approach introduces a lightweight sidecar Extractor with learnable compression tokens and a structured attention flow, enabling modular KV representations that can be flexibly reused and concatenated without modifying the frozen base model. We further employ a compression-concatenation co-training strategy to align extraction-time representations with their downstream reuse behavior. Extensive experiments across multiple long-context benchmarks and model families demonstrate that C$^2$KV significantly reduces KV cache storage and transfer costs, achieving up to 17$\times$ inference speedup under long contexts, while preserving generation quality.

Related papers

• S$^4$R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching [38.29032214906932]
  We propose S$4$R, which builds low-rank subspaces from selectively sampled tokens and computes attention over a sparsely reconstructed KV representation.<n>Experiments on LongBench and RULER with Llama and Qwen model families show that S$4$R achieves up to 5$times$ KV compression with near full-cache accuracy.
  arXiv  Detail & Related papers  (2026-08-01T08:41:29Z)
• EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction [55.026048429595384]
  EchoKV is a flexible KV cache compression scheme that enables on-demand transitions between standard and compressed inference.<n>We introduce a two-stage fine-tuning strategy that allows for rapid, low-cost training.
  arXiv  Detail & Related papers  (2026-03-24T07:58:42Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• KV Cache Transform Coding for Compact Storage in LLM Inference [2.20003167536462]
  We present KVTC, a lightweight transform coder that compresses KV caches for compact on- GPU and off- GPU storage.<n>By exploiting redundancies in KV caches, KVTC achieves up to 20$times$ compression while maintaining reasoning and long-context accuracy.<n>We test KVTC with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks including AIME25, LiveCodeBench, GSM8K, MMLU, Qasper, RULER, and MATH-500.
  arXiv  Detail & Related papers  (2025-11-03T18:20:35Z)
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction [37.97434606840326]
  Transformer-based large language models (LLMs) cache context as key-value ( KV) pairs during inference.<n>As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency.<n>This paper introduces KVzip, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries.
  arXiv  Detail & Related papers  (2025-05-29T13:05:47Z)
• FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management [48.904743679691414]
  FlowKV is a novel multi-turn isolation mechanism for KV Cache management.<n>It preserves the accumulated compressed KV cache from past turns.<n>It prevents the re-compression of older context and thereby mitigating catastrophic forgetting.
  arXiv  Detail & Related papers  (2025-05-21T10:20:46Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing [58.29726147780976]
  We propose a plug-and-play method called textit KVSharer, which shares the KV cache between layers to achieve layer-wise compression. Experiments show that textit KVSharer can reduce KV cache computation by 30%, thereby lowering memory consumption. We verify that textit KVSharer is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory.
  arXiv  Detail & Related papers  (2024-10-24T08:06:41Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios [13.144156413032896]
  We introduce CSKV, a training-efficient Channel Shrinking technique for KV cache compression. We show that CSKV can reduce the memory overhead of the KV cache by 80% while maintaining the model's long-context capability. Our method can be seamlessly combined with quantization to further reduce the memory overhead, achieving a compression ratio of up to 95%.
  arXiv  Detail & Related papers  (2024-09-16T17:36:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.