SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging

• URL: http://arxiv.org/abs/2603.14303v1
• Date: Sun, 15 Mar 2026 09:36:25 GMT
• Title: SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging
• Abstract summary: SemantiCache is a novel compression framework that preserves semantic integrity.<n>We first partition the cache into semantically coherent chunks.<n>Within each chunk, we introduce a Greedy Seed-Based Clustering (GSC) algorithm to group tokens into semantic clusters.<n>These clusters are further merged into semantic cores, enhanced by a Proportional Attention mechanism.
• Score: 14.82266992933174
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing KV cache compression methods generally operate on discrete tokens or non-semantic chunks. However, such approaches often lead to semantic fragmentation, where linguistically coherent units are disrupted, causing irreversible information loss and degradation in model performance. To address this, we introduce SemantiCache, a novel compression framework that preserves semantic integrity by aligning the compression process with the semantic hierarchical nature of language. Specifically, we first partition the cache into semantically coherent chunks by delimiters, which are natural semantic boundaries. Within each chunk, we introduce a computationally efficient Greedy Seed-Based Clustering (GSC) algorithm to group tokens into semantic clusters. These clusters are further merged into semantic cores, enhanced by a Proportional Attention mechanism that rebalances the reduced attention contributions of the merged tokens. Extensive experiments across diverse benchmarks and models demonstrate that SemantiCache accelerates the decoding stage of inference by up to 2.61 times and substantially reduces memory footprint, while maintaining performance comparable to the original model.

Related papers

• GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs [97.36238579001544]
  Large language models (LLMs) with extended context lengths rely on the key-value ( KV) cache to support attention over prior tokens.<n>Maintaining the KV cache incurs substantial memory overhead, motivating KV-cache compression methods.<n>We propose GRKV, a training-free KV-cache merging method that directly minimizes the discrepancy between compressed-cache and full-cache attention outputs.
  arXiv  Detail & Related papers  (2026-05-29T10:16:30Z)
• LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing [29.284917403504352]
  We propose LycheeCluster, a novel method for efficient KV cache management.<n>LycheeCluster preserves local semantic coherence via boundary-aware chunking and constructs a hierarchical index rooted in the triangle inequality.<n> Experiments demonstrate that LycheeCluster achieves up to a 3.6x end-to-end inference speedup with negligible degradation in model performance.
  arXiv  Detail & Related papers  (2026-03-09T14:50:35Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching [0.8307668828380427]
  We propose textitSemShareKV, a KV cache sharing and compression framework for large language models (LLMs)<n>Instead of relying on exact token matches, SemShareKV applies fuzzy token matching using locality-sensitive hashing (LSH) on token embeddings and incorporates Rotary Position Embedding (RoPE) to better preserve positional information.<n> Experiments on diverse summarization datasets show up to 6.25$times$ speedup and 42% lower GPU memory usage with 5k tokens input, with negligible quality degradation.
  arXiv  Detail & Related papers  (2025-09-29T14:16:13Z)
• KVCompose: Efficient Structured KV Cache Compression with Composite Tokens [7.922206020386125]
  Large language models (LLMs) rely on key-value (KV) caches for efficient autoregressive decoding.<n>We propose a simple, yet effective, KV cache compression framework based on attention-guided, layer-adaptive composite tokens.<n>Our method achieves significant memory reduction while preserving accuracy, consistently outperforming prior structured and semi-structured methods.
  arXiv  Detail & Related papers  (2025-09-05T14:58:24Z)
• CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing [54.34080239841088]
  CommonKV is a training-free method for cross-layer KV cache compression through adjacent parameters sharing.<n>We show that the proposed method consistently outperforms existing low-rank and cross-layer approaches at various compression ratios.
  arXiv  Detail & Related papers  (2025-08-22T06:55:45Z)
• SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching [5.545249777535219]
  SentenceKV is a novel KV caching approach designed to enhance inference efficiency while preserving semantic coherence.<n>We show that SentenceKV significantly outperforms state-of-the-art methods in both efficiency and memory usage, without compromising model accuracy.
  arXiv  Detail & Related papers  (2025-04-01T17:08:57Z)
• Hierarchical Semantic Compression for Consistent Image Semantic Restoration [62.97519327310638]
  We propose a novel hierarchical semantic compression (HSC) framework that purely operates within intrinsic semantic spaces from generative models.<n> Experimental results demonstrate that the proposed HSC framework achieves the state-of-the-art performance on subjective quality and consistency for human vision.
  arXiv  Detail & Related papers  (2025-02-24T03:20:44Z)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference [61.412894960600205]
  Large Language Models (LLMs) require significant GPU memory when processing long texts.<n>ChunkKV reimagines KV cache compression by treating semantic chunks as basic compression units.<n>Result: ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision.
  arXiv  Detail & Related papers  (2025-02-01T03:49:47Z)
• Efficient Inference of Vision Instruction-Following Models with Elastic Cache [76.44955111634545]
  We introduce Elastic Cache, a novel strategy for efficient deployment of instruction-following large vision-language models. We propose an importance-driven cache merging strategy to prune redundancy caches. For instruction encoding, we utilize the frequency to evaluate the importance of caches. Results on a range of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also notably outperforms existing pruning methods in language generation.
  arXiv  Detail & Related papers  (2024-07-25T15:29:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.