HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modeling

• URL: http://arxiv.org/abs/2606.08302v1
• Date: Sat, 06 Jun 2026 18:58:26 GMT
• Title: HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modeling
• Abstract summary: HACK++ is a training-free Head-Aware key-value Compression frameworK for Visual Autoregressive ( VAR) models.<n>It decouples attention from cache compression under independent budgets, bounding the current-scale attention cost while compressing the accumulated cache far more aggressively.<n>For example, on Infinity-2B/8B, HACK++ maintains near-lossless generation with only a 30% attention budget and a 10% cache budget, and remains robust even under a 1% cache budget.
• Score: 42.1403262611533
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Autoregressive (VAR) models adopt a next-scale prediction paradigm, offering high-quality generation with substantially fewer decoding steps. However, existing VAR models suffer from significant attention complexity and severe memory overhead due to the accumulation of key-value (KV) caches across scales. In this paper, we tackle this challenge by introducing KV cache compression into the next-scale paradigm. We begin with an in-depth analysis of VAR attention and observe that attention heads can be stably divided into two functionally distinct categories: Contextual Heads focus on maintaining semantic consistency, while Structural Heads preserve spatial coherence. Their functional divergence makes existing one-size-fits-all compression methods perform poorly on VAR models. We further find that the two head types differ markedly in their reliance on historical scales, and that this reliance shifts across layers and generation steps, arguing for an adaptive cache budget allocation. To address these challenges, we propose HACK++, a training-free Head-Aware key-value Compression frameworK for VAR models. From a one-time offline calibration, HACK++ classifies head types and derives head-specific priors. At inference, it decouples attention from cache compression under independent budgets, bounding the current-scale attention cost while compressing the accumulated cache far more aggressively, via pattern-specific strategies and a reliance-aware budget allocation. Extensive experiments on multiple VAR models across text-to-image, class-conditional, and unified understanding-and-generation tasks validate the effectiveness and generalizability of HACK++. For example, on Infinity-2B/8B, HACK++ maintains near-lossless generation with only a 30% attention budget and a 10% cache budget, and remains robust even under a 1% cache budget.

Related papers

• DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation [0.0]
  We propose an improved low-rank KV cache compression framework.<n>We show that our method reduces the number of Key cache parameters while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2026-07-27T12:08:56Z)
• Head-Aware Key-Value Compression for Efficient Autoregressive Image Generation [27.042998548651358]
  Autoregressive (AR) visual generation has achieved remarkable performance but suffers from high memory usage and low throughput.<n>Recent research has shown that retaining only a few lines of cache tokens can maintain high-quality images while significantly reducing memory usage and improving throughput.<n>We propose a novel head-aware key-value (KV) cache compression framework for autoregressive image generation, called HeadKV.
  arXiv  Detail & Related papers  (2026-05-20T01:30:33Z)
• HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference [14.17979669446161]
  We propose HeteroCache, a training-free dynamic compression framework.<n>We show that HeteroCache achieves state-of-the-art performance on multiple long-context benchmarks and accelerates decoding by up to $3times$ compared to the original model in the 224K context.
  arXiv  Detail & Related papers  (2026-01-20T07:35:06Z)
• Which Heads Matter for Reasoning? RL-Guided KV Cache Compression [15.865990296257413]
  Reasoning large language models exhibit complex reasoning behaviors through the extended chain-of-thought generation.<n>Existing KV cache compression methods underperform on reasoning models.<n>We propose RLKV, a novel reasoning-critical head identification framework.
  arXiv  Detail & Related papers  (2025-10-09T17:50:00Z)
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule [54.37983890753086]
  We introduce OjaKV, a framework that integrates a strategic hybrid storage policy with online subspace adaptation.<n>OjaKV preserves crucial first and most recent tokens in full-rank, maintaining high-fidelity anchors for attention.<n>It applies low-rank compression by incrementally adapting the projection basis using Oja's algorithm for online principal component analysis.
  arXiv  Detail & Related papers  (2025-09-25T21:42:27Z)
• CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing [54.34080239841088]
  CommonKV is a training-free method for cross-layer KV cache compression through adjacent parameters sharing.<n>We show that the proposed method consistently outperforms existing low-rank and cross-layer approaches at various compression ratios.
  arXiv  Detail & Related papers  (2025-08-22T06:55:45Z)
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration [69.57122277845293]
  We propose ReCalKV, a post-training low-rank KV cache compression approach with tailored strategies for Keys and Values.<n>For Keys, we propose Similarity aware Recontext (HSR), which clusters structurally similar heads into groups, enabling more accurate low-rank approximation.<n>For Values, we propose Offline Head-wise Value (OVC), which efficiently calibrates the value projection matrix using calibration data without training.
  arXiv  Detail & Related papers  (2025-05-30T08:49:27Z)
• Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling [27.79903885315292]
  Existing KV cache compression techniques for large language models are suboptimal for Visual Autoregressive ( VAR) models.<n>We propose HACK, a training-free Head-Aware Compression method for KV cache.
  arXiv  Detail & Related papers  (2025-04-12T15:42:17Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [97.41972925670508]
  Large vision-language models (LVLMs) incur significant computational and memory overhead during inference.<n>We present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• RazorAttention: Efficient KV Cache Compression Through Retrieval Heads [11.708388082001074]
  We propose a novel compression technique for Key-Value cache that preserves all token information. RazorAttention achieves a reduction in KV cache size by over 70% without noticeable impacts on performance.
  arXiv  Detail & Related papers  (2024-07-22T01:12:23Z)
• CORM: Cache Optimization with Recent Message for Large Language Model Inference [57.109354287786154]
  We introduce an innovative method for optimizing the KV cache, which considerably minimizes its memory footprint. CORM, a KV cache eviction policy, dynamically retains essential key-value pairs for inference without the need for model fine-tuning. Our validation shows that CORM reduces the inference memory usage of KV cache by up to 70% with negligible performance degradation across six tasks in LongBench.
  arXiv  Detail & Related papers  (2024-04-24T16:11:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.