TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

• URL: http://arxiv.org/abs/2605.17170v1
• Date: Sat, 16 May 2026 21:58:28 GMT
• Title: TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
• Abstract summary: We introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme for agentic workloads.<n>We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management.
• Score: 20.031258206130154
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agentic workloads have emerged as a major workload for LLM inference. They differ significantly from chat-only workloads, requiring long-context processing, the ability to handle multimodal inputs, and structured multi-turn interactions with tool calling capabilities. As a result, their context exhibits structure that can carry different importance along three key axes: temporal recency to the current turn, modality such as text or image tokens, and semantic role such as user queries, tool calls, observations, or reasoning. These axes capture distinct token behaviors and lead to different sensitivities to KV-cache compression. However, existing KV-cache quantization methods are typically homogeneous or exploit only heterogeneity on a single dimension, such as temporal proximity or modality, overlooking the interactions among them. To this end, we introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme that assigns each token a triaxial tag, calibrates per-tag sensitivity, and allocates INT2/INT4 bitwidths under a fixed memory budget. We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management, and custom fused Triton decode kernels. When using Qwen3-VL-32B-Thinking as a computer-use agent operating the OSWorld, TriAxialKV matches the accuracy of SGLang with BF16 KV cache while supporting 4.5$\times$ KV cache size and achieving 30% higher end-to-end throughput, when running on real GPU systems.

Related papers

• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration [67.17709877327232]
  Cate KV is a hybrid KV cache method that retains only critical token information for consistent heads.<n>Cate KV reduces memory usage by up to $2.72times$ and accelerates decoding by $2.18times$ in single-sample inputs.
  arXiv  Detail & Related papers  (2026-08-31T06:02:37Z)
• TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation [0.0]
  TriRoute is a single lightweight controller shared across all three axes.<n>It trains end-to-end via a heterogeneous relaxation.<n>It allocates full attention and high-precision cache to sentence-initial positions, rare subwords, and named entities.
  arXiv  Detail & Related papers  (2026-07-07T00:12:46Z)
• Tangram: Unlocking Non-Uniform KV Cache for Efficient Multi-turn LLM Serving [12.478756445519409]
  Tangram is a novel serving system designed to make Non-uniform KV caches practical.<n>Tangram improves throughput by up to 2.6x compared to existing baselines.
  arXiv  Detail & Related papers  (2026-06-04T15:41:27Z)
• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond [50.440302567029654]
  Multi-modal intelligence has made the Key-Value cache a dominant memory bottleneck for efficient deployment.<n>In this work, we revisit the inherent limitations of the per-channel quantization paradigm.<n>We propose OScaR, an accurate and lightweight KV cache compression framework for X-LLMs.
  arXiv  Detail & Related papers  (2026-05-19T10:53:03Z)
• SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving [42.268254679163725]
  KV-cache is a major bottleneck in real-world LLM serving.<n>Many KV-cache compression methods violate practical serving constraints.<n>We implement a fused rotation-quantization kernel that integrates directly into paged KV-cache layouts.
  arXiv  Detail & Related papers  (2026-04-21T07:12:23Z)
• InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models [4.4248984733976275]
  InnerQ is a hardware-aware KV-cache quantization scheme that decodes latency without sacrificing accuracy.<n>It applies group-wise quantization while grouping the cache matrices over their inner dimension.<n>Our evaluation experiments on Llama models shows that InnerQ maintains a few-shot GSM8K performance comparable to non-quantized KV caches.
  arXiv  Detail & Related papers  (2026-02-26T16:50:36Z)
• ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs [1.1267872663780352]
  Large Language Models (LLMs) are increasingly deployed in scenarios demanding ultra-long context reasoning.<n>Existing memory reduction techniques, including eviction and quantization, often rely on static cache.<n>We propose ARKV, a lightweight and adaptive framework that dynamically allocates precision levels to cached tokens.
  arXiv  Detail & Related papers  (2026-02-19T16:24:08Z)
• S$^3$-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference [11.779449360037518]
  We present S3-Attention, a memory-first inference-time framework that treats long-context processing as attention-aligned endogenous retrieval.<n>S3-Attention decodes transient key and query projections into top-k sparse feature identifiers using lightweight sparse autoencoders.<n>It constructs a CPU-based inverted index mapping features to token positions or spans during a single streaming scan.
  arXiv  Detail & Related papers  (2026-01-25T05:25:22Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression [54.28208936996186]
  Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks.<n> Quantization has emerged as a promising solution to reduce memory consumption while preserving historical information.<n>We propose XQuant, a training-free and plug-and-play framework that achieves ultra-low equivalent bit-width KV cache quantization.
  arXiv  Detail & Related papers  (2025-10-13T10:17:21Z)
• More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression [71.42818367729573]
  In large language models (LLMs), the memory usage of KV cache has become a critical bottleneck during inference.<n>The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension separately.<n>In this paper, we comprehensively investigate the token-precision trade-off in KV cache compression.
  arXiv  Detail & Related papers  (2024-12-17T09:20:31Z)
• DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction [33.936381781692994]
  DiffKV is a novel framework for efficient KV cache compression.<n>It exploits three levels of differentiation in the KV cache.<n>It is able to compress the KV cache by $2.7times$ to $5.7times$ with near-lossless accuracy.
  arXiv  Detail & Related papers  (2024-12-04T08:51:23Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.