vToken: Token-Level Virtualization for Reclaimable KV Caches

• URL: http://arxiv.org/abs/2608.13263v1
• Date: Thu, 13 Aug 2026 14:01:21 GMT
• Title: vToken: Token-Level Virtualization for Reclaimable KV Caches
• Abstract summary: PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation.<n>Recent KV eviction algorithms operate at a token granularity finer than block-level management.<n>We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement.
• Score: 18.919429120080277
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2\%--72.3\% and improves SLA-constrained throughput by up to 1.37$\times$. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2$\times$, while reducing the per-policy integration footprint from 500+ lines to under 50.

Related papers

• SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation [0.0]
  Large Language Models generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length.<n>Recent compression methods mitigate this cost via token merging.<n>We propose a training-free, dual-component framework for KV cache compression that addresses these limitations.
  arXiv  Detail & Related papers  (2026-05-19T17:44:56Z)
• Towards Joint Quantization and Token Pruning of Vision-Language Models [53.978753457744055]
  Token pruning and low-bit quantization are complementary for reducing inference costs.<n>We present a collaborative quantization-and-pruning framework that unifies low-bit inference and deterministic visual-token pruning.<n> Experiments on standard VLM benchmarks show improved robustness over stage-wise baselines under the same low-bit regime.
  arXiv  Detail & Related papers  (2026-04-19T08:18:29Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction [9.309829912599367]
  GraphKV is a graph-based framework that redefines token selection for KV cache compression.<n>It can be seamlessly utilized in existing KV cache eviction methods such as SnapKV and PyramidKV in a plug-and-play manner.
  arXiv  Detail & Related papers  (2025-08-30T06:56:28Z)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference [61.412894960600205]
  Large Language Models (LLMs) require significant GPU memory when processing long texts.<n>ChunkKV reimagines KV cache compression by treating semantic chunks as basic compression units.<n>Result: ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision.
  arXiv  Detail & Related papers  (2025-02-01T03:49:47Z)
• ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression [10.003118268356017]
  Long context poses significant challenges for inference efficiency.<n>We introduce ClusterKV, which recalls tokens at the granularity of semantic clusters.<n>Experiment results show that ClusterKV attains negligible accuracy loss across various tasks with 32k context lengths.
  arXiv  Detail & Related papers  (2024-12-04T10:58:27Z)
• Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity [24.118503938098307]
  textscPoD allocates memory according to token importance.<n>textscPoD reduces KV cache memory usage by up to 35% without compromising performance.
  arXiv  Detail & Related papers  (2024-12-03T08:29:27Z)
• LoCoCo: Dropping In Convolutions for Long Context Compression [77.26610232994508]
  This paper presents a novel approach, Dropping In Convolutions for Long Context Compression (LoCoCo) LoCoCo employs only a fixed-size Key-Value ( KV) cache, and can enhance efficiency in both inference and fine-tuning stages.
  arXiv  Detail & Related papers  (2024-06-08T01:35:11Z)
• Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference [78.65321721142624]
  We focus on a memory bottleneck imposed by the key-value ( KV) cache. Existing KV cache methods approach this problem by pruning or evicting large swaths of relatively less important KV pairs. We propose LESS, a simple integration of a constant sized cache with eviction-based cache methods.
  arXiv  Detail & Related papers  (2024-02-14T18:54:56Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.