SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

• URL: http://arxiv.org/abs/2604.21231v1
• Date: Thu, 23 Apr 2026 02:55:31 GMT
• Title: SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
• Abstract summary: SparKV is an adaptive KV loading framework that combines cloud-based KV streaming with on-device computation.<n>We show that SparKV reduces Time-to-First-Token by 1.3$x-5.1x with negligible impact on response quality, while lowering per-request energy consumption by 1.5x to 3.3x.
• Score: 8.037785679532316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Efficient inference for on-device Large Language Models (LLMs) remains challenging due to limited hardware resources and the high cost of the prefill stage, which processes the full input context to construct Key-Value (KV) caches. We present SparKV, an adaptive KV loading framework that combines cloud-based KV streaming with on-device computation. SparKV models the cost of individual KV chunks and decides whether each chunk should be streamed or computed locally, while overlapping the two execution paths to reduce latency. To handle fluctuations in wireless connectivity and edge resource availability, SparKV further refines offline-generated schedules at runtime to rebalance communication and computation costs. Experiments across diverse datasets, LLMs, and edge devices show that SparKV reduces Time-to-First-Token by 1.3$x-5.1x with negligible impact on response quality, while lowering per-request energy consumption by 1.5x to 3.3x, demonstrating its robustness and practicality for real-world on-device deployment.

Related papers

• C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference [31.428047145437773]
  Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning.<n>To mitigate the growing inference cost, recent work has explored key-value ( KV) cache reuse to reduce redundant prefill computation.<n>Existing reuse methods primarily focus on savings and overlook a critical bottleneck in long-context serving: the cost of storing and accessing KV caches.
  arXiv  Detail & Related papers  (2026-07-20T09:09:23Z)
• KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving [11.232399432774402]
  LLMs are widely adopted in production, pushing inference systems to their limits.<n>Disaggregated LLM serving (e.g., PD separation and KV state disaggregation) improves scalability and cost efficiency, but it also turns KV into an explicit payload crossing network and storage boundaries.<n>We present emphKVServe, the first service-aware and adaptive KV communication compression framework for disaggregated LLM serving.
  arXiv  Detail & Related papers  (2026-05-13T16:12:33Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs [13.800139639868883]
  We introduce a drift-robust, GPU-native KV-cache retrieval framework based on collision-based candidate selection.<n>ParisKV supports CPU-offloaded KV caches via Unified Virtual Addressing (UVA)<n>ParisKV matches or outperforms full attention quality on long-input and long-generation benchmarks.
  arXiv  Detail & Related papers  (2026-02-07T22:26:45Z)
• Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction [50.99402504483692]
  We propose a novel gating-based KV cache eviction method for frozen-weight language models.<n>Our approach integrates seamlessly into both the prefill and decoding stages.<n>Experiments show that our method maintains near-lossless performance while evicting up to 70% of the KV cache.
  arXiv  Detail & Related papers  (2026-01-25T03:07:54Z)
• XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization [58.92253769255316]
  LLM inference is challenging due to substantial memory footprint and bandwidth requirements.<n>XQuant exploits the rapidly increasing compute capabilities of hardware platforms to eliminate the memory bottleneck.<n>XQuant-CL exploits the cross-layer similarity in the X embeddings for extreme compression.
  arXiv  Detail & Related papers  (2025-08-14T06:52:38Z)
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models [77.84539432982307]
  We propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV)<n>R-KV preserves nearly 100% of the full KV cache performance using only 10% of the KV cache.<n>Remarkably, R-KV even achieves 105% of full KV cache performance with 16% of the KV cache.
  arXiv  Detail & Related papers  (2025-05-30T02:03:24Z)
• KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse [17.301289617498448]
  We present a KV cache management module that shares the KV cache across requests under multi-tenant scenarios.<n> KVShare reduces TTFT by up to 9.39x and increases 1.2x of the throughput compared to the full KV recompute.<n> KVShare achieves 20.38% boost in terms of accuracy compared to SOTA methods.
  arXiv  Detail & Related papers  (2025-03-17T16:43:35Z)
• DBudgetKV: Dynamic Budget in KV Cache Compression for Ensuring Optimal Performance [125.81664663201282]
  We introduce a new KV cache compression method dubbed DBudgetKV.<n>It features an attention-based metric to signal when the remaining KV cache is unlikely to match the full-cache performance.<n>Our method achieves lossless KV pruning effectively and robustly, exceeding 25% compression ratio on average.
  arXiv  Detail & Related papers  (2025-02-24T06:33:39Z)
• FastKV: KV Cache Compression for Fast Long-Context Processing with Token-Selective Propagation [14.33163594016033]
  Large language models (LLMs) require substantial prefill computation and key-value ( KV) cache.<n>Recent works that compress KV caches with prefill acceleration reduce this cost but inadvertently tie the prefill compute reduction to the decoding KV budget.<n>FastKV is a KV cache compression framework designed to reduce latency in both prefill and decoding by leveraging the stabilization of token importance in later layers.
  arXiv  Detail & Related papers  (2025-02-03T05:25:09Z)
• DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction [33.936381781692994]
  DiffKV is a novel framework for efficient KV cache compression.<n>It exploits three levels of differentiation in the KV cache.<n>It is able to compress the KV cache by $2.7times$ to $5.7times$ with near-lossless accuracy.
  arXiv  Detail & Related papers  (2024-12-04T08:51:23Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.