CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

• URL: http://arxiv.org/abs/2602.08686v1
• Date: Mon, 09 Feb 2026 14:07:55 GMT
• Title: CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation
• Abstract summary: We propose CompilerKV, a risk-adaptive and head-aware compression framework for large language models.<n>CoherentKV compiles offline experience into reusable decision tables for prefill-only deployment.<n>Experiments on LongBench show CompilerKV dominates SOTA methods under a 512-token budget, recovering 97.7% of FullKV performance while achieving up to +5.2 points gain over the strongest competitor.
• Score: 16.540537092341737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) in long-context scenarios are severely constrained by the linear growth of Key-Value (KV) cache memory. Existing KV compression methods rely either on static thresholds and attention-only heuristics or on coarse memory budget allocation. Under tight memory budgets, these methods overlook two key factors: prompt-dependent variation in compression risk and functional heterogeneity across attention heads, which destabilize token selection and lead to tail failures. To address these challenges, we propose CompilerKV, a risk-adaptive and head-aware compression framework that compiles offline experience into reusable decision tables for prefill-only deployment. CompilerKV integrates two key synergistic components: (i) a Head Heterogeneity Table, learned via offline contextual bandits, which assigns head-specific reliability weights to govern functional differences across attention heads explicitly; and (ii) a Risk-Adaptive Threshold Gating mechanism that jointly models attention entropy and local perplexity, transforming prompt-level risk into deployable retention thresholds. Experiments on LongBench show CompilerKV dominates SOTA methods under a 512-token budget, recovering 97.7\% of FullKV performance while achieving up to +5.2 points gain over the strongest competitor.

Related papers

• HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression [11.40656413331368]
  Long-context LLM inference faces a fundamental conflict: head-adaptive compression algorithms offer superior accuracy by dynamically fluctuating memory budgets.<n>Yet modern inference engines demand rigid, static memory patterns to leverage Graphs and PagedAttention.<n>We resolve this Static-Dynamic'' mismatch with HARD-KV, a unified framework that bridges dynamic selection with rigid system constraints.
  arXiv  Detail & Related papers  (2026-06-27T09:36:37Z)
• HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modeling [42.1403262611533]
  HACK++ is a training-free Head-Aware key-value Compression frameworK for Visual Autoregressive ( VAR) models.<n>It decouples attention from cache compression under independent budgets, bounding the current-scale attention cost while compressing the accumulated cache far more aggressively.<n>For example, on Infinity-2B/8B, HACK++ maintains near-lossless generation with only a 30% attention budget and a 10% cache budget, and remains robust even under a 1% cache budget.
  arXiv  Detail & Related papers  (2026-06-06T18:58:26Z)
• Adaptive Mass-Segmented KV Compression for Long-Context Reasoning [7.152154762874602]
  We propose Adaptive Mass-Segmented (AMS) KV Compression, a framework that shifts the paradigm from token-level competition to region-aware quota allocation.<n>AMS adaptively partitions the KV cache based on the spatial distribution of attention mass, ensuring structurally vital reasoning segments receive guaranteed memory quotas.<n>It can be seamlessly integrated into representative methods such as TOVA, Expected Attention, KeyDiff, R-KV and Tri.Attention.
  arXiv  Detail & Related papers  (2026-05-22T03:32:52Z)
• EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction [55.026048429595384]
  EchoKV is a flexible KV cache compression scheme that enables on-demand transitions between standard and compressed inference.<n>We introduce a two-stage fine-tuning strategy that allows for rapid, low-cost training.
  arXiv  Detail & Related papers  (2026-03-24T07:58:42Z)
• Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics [22.98826013817833]
  We propose a physics-inspired view of KV compression as a controlled perturbation of token-level routing.<n>We find that moderate compression degrades internal representations with little accuracy loss, revealing redundancy.<n>We identify representational rigidity, where excessive head-level consensus collapses routing flexibility despite token survival.
  arXiv  Detail & Related papers  (2026-03-02T04:16:36Z)
• AHC: Meta-Learned Adaptive Compression for Continual Object Detection on Memory-Constrained Microcontrollers [0.14504054468850666]
  Continual object detection on microcontrollers (MCUs) with under 100KB memory requires efficient feature compression.<n>We introduce Adaptive Hierarchical Compression (AHC), a meta-learning framework featuring three key innovations.<n>AHC enables practical continual detection within a 100KB replay budget, achieving competitive accuracy through mean-pooled compressed feature replay.
  arXiv  Detail & Related papers  (2026-02-24T07:06:51Z)
• Low-Rank Key Value Attention [3.7728602841318426]
  Transformer pretraining is increasingly constrained by memory and compute requirements.<n>We propose textitlow-rank KV adaptation (LRKV), a simple modification of multi-head attention.<n>LRKV is a drop-in replacement for standard multi-head attention.
  arXiv  Detail & Related papers  (2026-01-16T17:56:40Z)
• Training-free Context-adaptive Attention for Efficient Long Context Modeling [57.703159205740185]
  Training-free Context-adaptive Attention (TCA-Attention) is a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference.<n>TCA-Attention achieves a 2.8$times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention.
  arXiv  Detail & Related papers  (2025-12-10T01:54:57Z)
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule [54.37983890753086]
  We introduce OjaKV, a framework that integrates a strategic hybrid storage policy with online subspace adaptation.<n>OjaKV preserves crucial first and most recent tokens in full-rank, maintaining high-fidelity anchors for attention.<n>It applies low-rank compression by incrementally adapting the projection basis using Oja's algorithm for online principal component analysis.
  arXiv  Detail & Related papers  (2025-09-25T21:42:27Z)
• KVCompose: Efficient Structured KV Cache Compression with Composite Tokens [7.922206020386125]
  Large language models (LLMs) rely on key-value (KV) caches for efficient autoregressive decoding.<n>We propose a simple, yet effective, KV cache compression framework based on attention-guided, layer-adaptive composite tokens.<n>Our method achieves significant memory reduction while preserving accuracy, consistently outperforming prior structured and semi-structured methods.
  arXiv  Detail & Related papers  (2025-09-05T14:58:24Z)
• Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization [27.994376063677766]
  Transformer models face scalability challenges due to inefficient memory allocation for growing key-value caches.<n>We propose mixSGA, a novel mixture-of-expert approach that dynamically optimize token-wise computation and memory allocation.<n>Our key novelties include: (1) a token-wise expert-choice routing mechanism guided by learned importance scores; (2) weight-sharing across grouped attention projections to minimize parameter overhead; and (3) an auxiliary loss to ensure one-hot routing decisions for training-inference consistency in CLMs.
  arXiv  Detail & Related papers  (2025-06-16T14:30:17Z)
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration [69.57122277845293]
  We propose ReCalKV, a post-training low-rank KV cache compression approach with tailored strategies for Keys and Values.<n>For Keys, we propose Similarity aware Recontext (HSR), which clusters structurally similar heads into groups, enabling more accurate low-rank approximation.<n>For Values, we propose Offline Head-wise Value (OVC), which efficiently calibrates the value projection matrix using calibration data without training.
  arXiv  Detail & Related papers  (2025-05-30T08:49:27Z)
• Runtime Adaptive Pruning for LLM Inference [7.5252252615137225]
  We propose RAP, an elastic pruning framework driven by reinforcement learning (RL)<n>RAP tracks the evolving ratio between model parameters and KV-cache across practical execution.<n>RAP outperforms state-of-the-art baselines, marking the first time to jointly consider model weights and KVcache on the fly.
  arXiv  Detail & Related papers  (2025-05-22T06:12:42Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective [85.08718140718707]
  UNComp is an uncertainty-aware framework that uncovers sparsity patterns that can be used for adaptive compression.<n>By focusing on uncertainty to analyze the sparsity pattern in detail, UNComp reduces the KV cache size to 4.74% of the original, achieves a 6% prefill speedup, and improves throughput by 6.4x.
  arXiv  Detail & Related papers  (2024-10-04T02:32:36Z)
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs [82.08922896531618]
  We introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs) We conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens.
  arXiv  Detail & Related papers  (2023-10-03T05:17:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.