KVShare: Semantic-Aware Key-Value Cache Sharing for Efficient Large Language Model Inference

• URL: http://arxiv.org/abs/2503.16525v1
• Date: Mon, 17 Mar 2025 16:43:35 GMT
• Title: KVShare: Semantic-Aware Key-Value Cache Sharing for Efficient Large Language Model Inference
• Authors: Huan Yang, Renji Zhang, Deyu Zhang,
• Abstract summary: KVShare is a multi-user Key-Value ( KV) Cache sharing technology based on semantic similarity.<n>It is designed to enhance the inference efficiency of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs)
• Score: 7.894452711850396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents KVShare, a multi-user Key-Value (KV) Cache sharing technology based on semantic similarity, designed to enhance the inference efficiency of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs). Addressing the limitations of existing prefix caching (strict text prefix matching) and semantic caching (loss of response diversity), KVShare achieves fine-grained KV cache reuse through semantic alignment algorithms and differential editing operations. Experiments on real-world user conversation datasets demonstrate that KVShare improves KV cache hit rates by over 60%, while maintaining output quality comparable to full computation (no significant degradation in BLEU and Rouge-L metrics). This approach effectively reduces GPU resource consumption and is applicable to scenarios with repetitive queries, such as healthcare and education.

Related papers

• Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs? [79.58770714228983]
  Language models handle increasingly long contexts for tasks such as book summarization.<n>This leads to growing memory costs for the key-value ( KV) cache.<n>Many prior works have proposed ways of discarding KVs from memory, but their approaches are tailored to favorable settings.<n>We propose the * KV footprint* as a unified metric, which accounts for both the amount of KV entries stored and their lifespan in memory.
  arXiv  Detail & Related papers  (2025-06-20T16:21:12Z)
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models [77.84539432982307]
  We propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV)<n>R-KV preserves nearly 100% of the full KV cache performance using only 10% of the KV cache.<n>Remarkably, R-KV even achieves 105% of full KV cache performance with 16% of the KV cache.
  arXiv  Detail & Related papers  (2025-05-30T02:03:24Z)
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction [37.489744618357655]
  Transformer-based large language models (LLMs) cache context as key-value ( KV) pairs during inference.<n>As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency.<n>This paper introduces KVzip, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries.
  arXiv  Detail & Related papers  (2025-05-29T13:05:47Z)
• FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference [14.592018362921875]
  FreeKV is an algorithm-system co-optimization framework to enhance KV retrieval efficiency while preserving accuracy.<n>Experiments demonstrate that FreeKV achieves near-lossless accuracy across various scenarios and models, delivering up to 13$times$ speedup.
  arXiv  Detail & Related papers  (2025-05-19T13:36:45Z)
• LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference [16.83202690345235]
  We propose Self-Attention Guided Eviction(SAGE-KV), a simple and effective KV eviction cache method for long-context inference.<n>After prefilling, our method performs a one-time top-k selection at both the token and head levels to compress the KV cache.<n>SAGE-KV achieves 4x higher memory efficiency with improved accuracy over the static KV cache selection method StreamLLM, and 2x higher memory efficiency with better accuracy than the dynamic KV cache selection method Quest.
  arXiv  Detail & Related papers  (2025-03-11T20:45:02Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance, then halting the pruning process.<n>Our method is easy to integrate within LLM inference, not only optimizing memory space, but also showing reduced inference time compared to existing methods.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse [35.97391418064724]
  KVLink is an approach for efficient key-value ( KV) cache reuse in large language models (LLMs)<n> KVLink introduces three key components: adjusting positional embeddings of KV cache at inference to match the global position after concatenation, using trainable special tokens to restore self-attention, and applying mixed-data fine-tuning.<n> Experiments across 7 datasets demonstrate that KVLink improves question answering accuracy by an average of 4% over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-02-21T23:34:29Z)
• QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache [67.84112700032007]
  Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings.<n>In these scenarios, the Key-Value ( KV) cache is the primary bottleneck in terms of both GPU memory and latency.<n>We propose a novel self-speculative decoding framework, QuantSpec, where the draft model shares the architecture of the target model but employs a hierarchical 4-bit quantized KV cache and 4-bit quantized weights for acceleration.
  arXiv  Detail & Related papers  (2025-02-05T20:43:48Z)
• SCBench: A KV Cache-Centric Analysis of Long-Context Methods [61.025422435235456]
  We introduce SCBench, a benchmark for evaluating long-context methods from a KV cachecentric perspective.<n>We provide an extensive KV cache-centric analysis of eight categories long-context solutions, including Gated Linear RNNs and Mamba-Attention hybrids.<n>Our findings show that sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding with O(n) memory and sub-O(n2) pre-filling perform robustly.
  arXiv  Detail & Related papers  (2024-12-13T17:59:52Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [65.36715026409873]
  Key-value (KV) cache, necessitated by the lengthy input and output sequences, notably contributes to the high inference cost.<n>We present PrefixKV, which reframes the challenge of determining KV cache sizes for all layers into the task of searching for the optimal global prefix configuration.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression [10.003118268356017]
  Long context poses significant challenges for inference efficiency.<n>We introduce ClusterKV, which recalls tokens at the granularity of semantic clusters.<n>Experiment results show that ClusterKV attains negligible accuracy loss across various tasks with 32k context lengths.
  arXiv  Detail & Related papers  (2024-12-04T10:58:27Z)
• A Method for Building Large Language Models with Predefined KV Cache Capacity [11.710667043543545]
  The Bounded-Cache Transformer (BCT) addresses the excessive memory consumption issue in traditional KV caches.<n>By dynamically updating the key-value vector sequences, the BCT achieves efficient inference within limited cache capacity.<n> Experimental results demonstrate that the BCT significantly reduces memory usage while maintaining the model's inference quality.
  arXiv  Detail & Related papers  (2024-11-24T11:30:00Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption [66.97998742151918]
  Large Language Models (LLMs) have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture's struggle with handling long texts. KV Cache has emerged as a pivotal solution, converting the time complexity of token generation from quadratic to linear.
  arXiv  Detail & Related papers  (2024-07-25T12:56:22Z)
• LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference [32.20654044142376]
  LOOK-M is a pioneering, fine-tuning-free approach that efficiently reduces the multimodal KV cache size. It achieves up to 1.5x faster decoding and also maintains or even enhances performance across a variety of long context multimodal tasks.
  arXiv  Detail & Related papers  (2024-06-26T07:44:24Z)
• SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models [43.22490117833939]
  SKVQ stands for sliding-window KV cache quantization. S KVQ rearranges the channels of the KV cache in order to improve the similarity of channels in quantization groups. It is possible to process context lengths of up to 1M on an 80GB memory GPU for a 7b model and up to 7 times faster decoding.
  arXiv  Detail & Related papers  (2024-05-10T03:06:24Z)
• CORM: Cache Optimization with Recent Message for Large Language Model Inference [57.109354287786154]
  We introduce an innovative method for optimizing the KV cache, which considerably minimizes its memory footprint. CORM, a KV cache eviction policy, dynamically retains essential key-value pairs for inference without the need for model fine-tuning. Our validation shows that CORM reduces the inference memory usage of KV cache by up to 70% with negligible performance degradation across six tasks in LongBench.
  arXiv  Detail & Related papers  (2024-04-24T16:11:54Z)
• QAQ: Quality Adaptive Quantization for LLM KV Cache [3.163526369095745]
  A bottleneck in model deployment emerges due to the linear expansion of the Key-Value cache with the context length. We propose QAQ, a Quality Adaptive Quantization scheme for the KV cache.
  arXiv  Detail & Related papers  (2024-03-07T16:42:37Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.