CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention

• URL: http://arxiv.org/abs/2603.17946v1
• Date: Wed, 18 Mar 2026 17:18:35 GMT
• Title: CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attention
• Abstract summary: We propose a Covariance-Aware, Rank-Enhanced MLA conversion pipeline under a fixed KV width.<n> CARE introduces three key steps: (i) activation-preserving factorization, which aligns the approximation with the actual input activations rather than just the weights; (ii) adjusted-rank allocation, which spreads a fixed KV budget across layers by giving more capacity to layers that need it most; and (iii) KV-parity mapping, which re parameterizes the converted K and V to fit the MLA format while keeping the KV-cache size unchanged.
• Score: 35.44699837487632
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Converting pretrained attention modules such as grouped-query attention (GQA) into multi-head latent attention (MLA) can improve expressivity without increasing KV-cache cost, making it attractive for efficient inference. However, many practical conversion baselines rely on weight-only low-rank approximations (e.g., SVD-style initializations) and uniform rank allocation. They focus on minimizing the difference between weight matrices rather than on how those weights affect input activations, ignore the covariance structure of activations, and enforce uniform rank across layers, causing activation drift and degraded attention fidelity. To address these issues, we propose CARE, a Covariance-Aware, Rank-Enhanced MLA conversion pipeline under a fixed KV width. CARE introduces three key steps: (i) activation-preserving factorization, which aligns the approximation with the actual input activations rather than just the weights; (ii) adjusted-rank allocation, which spreads a fixed KV budget across layers by giving more capacity to layers that need it most; and (iii) KV-parity mapping, which reparameterizes the converted K and V to fit the MLA format while keeping the KV-cache size unchanged. Our method outperforms a uniform-rank SVD baseline on Qwen3-4B/30B-A3B-Instruct-2507 and Llama-3.1-8B/70B-Instruct, reducing one-shot perplexity by up to 215x and improving mean accuracy by up to 1.70x at matched KV budgets. With a brief post-SVD healing fine-tune, we fully recover the original model's accuracy.

Related papers

• STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control [8.78945073362251]
  STAR-KV is an adaptive low-rank KV cache compression framework with fine-grained rank control.<n> evaluated across multiple LLMs and benchmarks.<n>Enabled by custom Triton-based GPU kernels, STAR-KV delivers up to 6.9x speedup for the attention module and 3.1x end-to-end generation throughput.
  arXiv  Detail & Related papers  (2026-06-07T00:24:02Z)
• KVSculpt: KV Cache Compression as Distillation [7.085426079187912]
  KV cache compression is critical for efficient long-context LLM inference.<n>Existing methods range from pure eviction -- selecting which KV pairs to keep -- to merging, which combines similar pairs into fewer ones.<n>We propose KVSculpt, which moves to the other end of this spectrum.<n>Keys are optimized via L-BFGS and values are solved in closed form via least squares.
  arXiv  Detail & Related papers  (2026-03-29T19:14:25Z)
• ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution [84.41751286055909]
  We develop a training-based KV cache eviction framework that learns to predict which KV pairs to evict during longtext generations.<n>We formulate cache eviction as a Markov Decision Process and apply the GRPO algorithm to mitigate the significant language modeling loss increase on low-entropy tokens.
  arXiv  Detail & Related papers  (2026-02-03T07:16:51Z)
• Low-Rank Key Value Attention [3.7728602841318426]
  Transformer pretraining is increasingly constrained by memory and compute requirements.<n>We propose textitlow-rank KV adaptation (LRKV), a simple modification of multi-head attention.<n>LRKV is a drop-in replacement for standard multi-head attention.
  arXiv  Detail & Related papers  (2026-01-16T17:56:40Z)
• AMS-KV: Adaptive KV Caching in Multi-Scale Visual Autoregressive Transformers [6.1675897118034975]
  Key and Value ( KV) caching in large language models (LLMs) has been extensively studied, but next-scale prediction presents unique challenges.<n>We introduce AMS-KV, a scale-adaptive KV caching policy for next-scale prediction in VAR models.<n>Compared to the vanilla next-scale prediction-based VAR models, AMS-KV reduces KV cache usage by up to 84.83% and self-attention latency by 60.48%.
  arXiv  Detail & Related papers  (2025-11-20T05:10:12Z)
• PatternKV: Flattening KV Representation Expands Quantization Headroom [37.83913102876393]
  KV cache in autoregressive LLMs eliminates redundant recomputation but has emerged as the dominant memory and bandwidth bottleneck during inference.<n> KV quantization is a key lever for reducing cache cost, but accuracy drops sharply as the native KV distribution lacks flatness.<n>We show that the K cache maintains a stable structure that evolves gradually with context, while the V cache carries latent semantic regularities.
  arXiv  Detail & Related papers  (2025-10-05T12:09:14Z)
• SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference [71.20542521694524]
  SmallKV is a small model assisted compensation method for KV cache compression.<n>We show that SmallKV achieves 1.75 - 2.56 times higher throughput than baseline methods.
  arXiv  Detail & Related papers  (2025-08-03T09:15:36Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• Effectively Compress KV Heads for LLM [28.0801697946958]
  We propose a novel approach for compressing Key-Value ( KV) caches. Our method can compress half or even three-quarters of KV heads while maintaining performance comparable to the original LLMs.
  arXiv  Detail & Related papers  (2024-06-11T08:37:33Z)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization [67.74400574357472]
  LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference.<n> Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision.<n>Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods.
  arXiv  Detail & Related papers  (2024-01-31T18:58:14Z)
• ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models [49.970828419830355]
  We introduce a new post-training compression paradigm for Large Language Models (LLMs)<n>We propose a training-free approach called Activation-aware Singular Value Decomposition (ASVD)
  arXiv  Detail & Related papers  (2023-12-10T08:41:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.