A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs

• URL: http://arxiv.org/abs/2607.12550v2
• Date: Fri, 17 Jul 2026 17:45:07 GMT
• Title: A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs
• Abstract summary: Key-value (KV) cache has become the dominant memory cost of transformer inference.<n>Low-rank methods factor two-dimensional slices of the cache, or per-head matrices or cross-layer feature blocks, and quantization methods lower the bit-width of every entry.<n>Our method, JoLT, applies a partial Tucker decomposition that compresses only the token and feature axes while leaving the head and layer axes intact.<n>A randomized-SVD variant, FlashJoLT, delivers a 5-13x compression-time speedup at 1024-token context and matched quality.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference: it grows with batch size, context length, and depth, and at long context it, rather than the model weights, sets the throughput ceiling. Existing reductions fall into two families. Low-rank methods factor two-dimensional slices of the cache, either per-head matrices or cross-layer feature blocks, and quantization methods lower the bit-width of every entry. Neither exploits the fact that the cache at a layer is naturally a third-order tensor whose three axes, the heads, the tokens, and the features, carry very different amounts of redundancy. We take this tensor view directly. Our method, JoLT (Joint Lagrangian Tucker), applies a partial Tucker decomposition that compresses only the token and feature axes while leaving the head and layer axes intact, then restores the energy that truncation discards with a rotated low-bit residual: a random orthogonal rotation followed by low-bit quantization. A single Lagrangian dual allocates the Tucker ranks and the residual bit-widths together, per layer group and separately for keys and values, under one byte budget. The result is a near-lossless 2-3x compression. Perplexity stays near-lossless on both a grouped-query-attention model (Mistral-7B-v0.3) and a multi-head-attention model (LLaMA-2-13B), and GSM8K accuracy and needle-in-a-haystack retrieval hold at the uncompressed baseline at 2x on both architectures and through 3x on the GQA model. At 2x, JoLT reconstructs the cache to relative Frobenius error 0.009 (K) and 0.006 (V) on both architectures. A randomized-SVD variant, FlashJoLT, delivers a 5-13x compression-time speedup at 1024-token context and matched quality.

Related papers

• Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers [4.632241550169362]
  We introduce Looped Latent Attention (LLA), a post-training cache that stores compact K and V latents and reconstructs loop-specific K/V vectors only when attention reads them.<n>At matched cache budget, per-head LLA outperforms head-axis MLA, cross-layer sharing, KV quantization and final-loop reuse.<n>The same advantage holds on Ouro-2.6B-Thinking and transfers to Huginn-3.5B, where an axis SVD remains near-lossless to 32x in collapsible evaluation.
  arXiv  Detail & Related papers  (2026-07-16T20:58:16Z)
• OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantization [13.284869342523095]
  Key-value (KV) dominates memory bandwidth and footprint in autoregressive inference.<n>Recent rotation-preconditioned codecs (TurboQuant, PolarQuant) show that a structured random rotation followed by a per-coordinate quantizer matched to an analytically tractable marginal is a near-optimal recipe for KV compression.<n>OCTOPUS advances this paradigm through joint quantization of rotated coordinate triplets.
  arXiv  Detail & Related papers  (2026-05-20T14:19:51Z)
• MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression [7.208745673318648]
  Existing compression methods each act on a single axis of the four-dimensional KV tensor.<n>We show that different layers respond very differently to each compression operation.<n>We propose MoE-nD, a mixture-of-experts framework that routes each layer to its own.
  arXiv  Detail & Related papers  (2026-04-20T01:20:26Z)
• Sequential KV Cache Compression via Probabilistic Language Tries: Beyond the Per-Vector Shannon Limit [0.0]
  Recent work on KV cache quantization, culminating in TurboQuant, has approached the Shannon entropy limit for per-vector compression of transformer key-value caches.<n>We observe that this limit applies to a strictly weaker problem than the one that actually matters: compressing the KV cache as a sequence.<n>The tokens stored in a KV cache are not arbitrary floating-point data -- they are samples from the exact formal language the model was trained on, and the model is by construction a near-optimal predictor of that language.
  arXiv  Detail & Related papers  (2026-04-10T22:48:19Z)
• KVSculpt: KV Cache Compression as Distillation [7.085426079187912]
  KV cache compression is critical for efficient long-context LLM inference.<n>Existing methods range from pure eviction -- selecting which KV pairs to keep -- to merging, which combines similar pairs into fewer ones.<n>We propose KVSculpt, which moves to the other end of this spectrum.<n>Keys are optimized via L-BFGS and values are solved in closed form via least squares.
  arXiv  Detail & Related papers  (2026-03-29T19:14:25Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression [54.28208936996186]
  Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks.<n> Quantization has emerged as a promising solution to reduce memory consumption while preserving historical information.<n>We propose XQuant, a training-free and plug-and-play framework that achieves ultra-low equivalent bit-width KV cache quantization.
  arXiv  Detail & Related papers  (2025-10-13T10:17:21Z)
• TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference [48.40143137402824]
  Multi-Head Latent Attention (MLA) compresses key-value states into a low-rank latent vector, caching only this vector to reduce memory.<n>In tensor parallelism (TP), however, attention heads are computed across multiple devices, and each device must load the full cache.<n>We propose-Parallel Latent Attention (TPLA), a scheme that partitions both the latent representation and each head's input dimension across devices, performs attention independently per shard, and then combines results with an all-reduce.
  arXiv  Detail & Related papers  (2025-08-21T15:25:40Z)
• ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration [69.57122277845293]
  We propose ReCalKV, a post-training low-rank KV cache compression approach with tailored strategies for Keys and Values.<n>For Keys, we propose Similarity aware Recontext (HSR), which clusters structurally similar heads into groups, enabling more accurate low-rank approximation.<n>For Values, we propose Offline Head-wise Value (OVC), which efficiently calibrates the value projection matrix using calibration data without training.
  arXiv  Detail & Related papers  (2025-05-30T08:49:27Z)
• KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing [58.29726147780976]
  We propose a plug-and-play method called textit KVSharer, which shares the KV cache between layers to achieve layer-wise compression. Experiments show that textit KVSharer can reduce KV cache computation by 30%, thereby lowering memory consumption. We verify that textit KVSharer is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory.
  arXiv  Detail & Related papers  (2024-10-24T08:06:41Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM [37.87634266742105]
  Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. Existing methods rely on dropping unimportant tokens or quantizing all entries uniformly. We propose GEAR, an efficient KV cache compression framework that achieves near-lossless high-ratio compression.
  arXiv  Detail & Related papers  (2024-03-08T18:48:30Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.