AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor

• URL: http://arxiv.org/abs/2606.17872v1
• Date: Tue, 16 Jun 2026 12:43:18 GMT
• Title: AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor
• Abstract summary: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks.<n>Their large size introduces significant challenges in memory usage, energy cost, and on-device deployment.<n>We propose AnchorKV, a drop-in modification to KV cache compression that biases token retention scores away from directions in key space associated with harmful prompts.
• Score: 20.7510970936151
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment. Since scaling pre-trained language models improves downstream capability \cite{zhao2023survey}, the key-value (KV) cache becomes a dominant inference bottleneck. Recent KV cache compression methods \cite{jo2025fastkv,li2024snapkv,zhou2024dynamickv} reduce this cost by retaining only a subset of attention-relevant tokens. However, while these approaches preserve accuracy on benign workloads, their compression policies either fail to defend against jailbreak attacks \cite{jiang2024robustkv} or degrade safety alignment under aggressive eviction. We propose AnchorKV, a drop-in modification to KV cache compression that biases token retention scores away from directions in key space associated with harmful prompts. AnchorKV constructs an offline safety anchor by adapting a difference-of-means representation engineering approach \cite{arditi2024refusal,zou2023representation} to the layer-specific key projection space used in KV caching. Based on this anchor, a soft penalty token selection rule trades a small amount of utility for substantially improved safety alignment, while reducing to the original compressor when the penalty is zero.

Related papers

• AnchorKV: Anchor-Residual KV Cache Compression [11.048086031066623]
  We propose AnchorKV, a compression scheme that shrinks the cache by $20times$ without discarding a single token.<n>AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale.
  arXiv  Detail & Related papers  (2026-08-03T21:38:30Z)
• GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs [97.36238579001544]
  Large language models (LLMs) with extended context lengths rely on the key-value ( KV) cache to support attention over prior tokens.<n>Maintaining the KV cache incurs substantial memory overhead, motivating KV-cache compression methods.<n>We propose GRKV, a training-free KV-cache merging method that directly minimizes the discrepancy between compressed-cache and full-cache attention outputs.
  arXiv  Detail & Related papers  (2026-05-29T10:16:30Z)
• KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression [3.9838043625801967]
  Reasoning language models generate long chain-of-thought (CoT)<n>CoT accumulates a massive KV cache during the decoding phase.<n> KV cache compression has emerged as a promising technique for reducing memory overhead.
  arXiv  Detail & Related papers  (2026-05-01T15:54:13Z)
• KVReviver: Reversible KV Cache Compression with Sketch-Based Token Reconstruction [20.53279247581787]
  We propose KVReviver, a reversible KV cache compression method based on the sketch algorithm.<n>In 2k-length contexts, it requires only 10% of KV Cache budget while maintaining identical end-to-end inference accuracy.<n>For 32k-length contexts, it achieves equivalent or comparable accuracy 2% accuracy loss.
  arXiv  Detail & Related papers  (2025-12-01T03:59:20Z)
• SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression [7.603859408568262]
  Large Language Models (LLMs) face a significant bottleneck during autoregressive inference due to the massive memory footprint of the Key-Value (KV) cache.<n>We introduce SWAN, a novel, fine-tuning-free framework that eliminates this overhead.<n>Our method uses an offline matrix to rotate and prune the KV-cache, which is then used directly in the attention computation without any reconstruction.
  arXiv  Detail & Related papers  (2025-11-24T09:41:24Z)
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule [54.37983890753086]
  We introduce OjaKV, a framework that integrates a strategic hybrid storage policy with online subspace adaptation.<n>OjaKV preserves crucial first and most recent tokens in full-rank, maintaining high-fidelity anchors for attention.<n>It applies low-rank compression by incrementally adapting the projection basis using Oja's algorithm for online principal component analysis.
  arXiv  Detail & Related papers  (2025-09-25T21:42:27Z)
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration [69.57122277845293]
  We propose ReCalKV, a post-training low-rank KV cache compression approach with tailored strategies for Keys and Values.<n>For Keys, we propose Similarity aware Recontext (HSR), which clusters structurally similar heads into groups, enabling more accurate low-rank approximation.<n>For Values, we propose Offline Head-wise Value (OVC), which efficiently calibrates the value projection matrix using calibration data without training.
  arXiv  Detail & Related papers  (2025-05-30T08:49:27Z)
• FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management [48.904743679691414]
  FlowKV is a novel multi-turn isolation mechanism for KV Cache management.<n>It preserves the accumulated compressed KV cache from past turns.<n>It prevents the re-compression of older context and thereby mitigating catastrophic forgetting.
  arXiv  Detail & Related papers  (2025-05-21T10:20:46Z)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference [61.412894960600205]
  Large Language Models (LLMs) require significant GPU memory when processing long texts.<n>ChunkKV reimagines KV cache compression by treating semantic chunks as basic compression units.<n>Result: ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision.
  arXiv  Detail & Related papers  (2025-02-01T03:49:47Z)
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs [82.08922896531618]
  We introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs) We conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens.
  arXiv  Detail & Related papers  (2023-10-03T05:17:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.