HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

• URL: http://arxiv.org/abs/2606.21633v1
• Date: Fri, 19 Jun 2026 17:36:51 GMT
• Title: HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval
• Abstract summary: Diffusion LLMs (dLLMs) improve GPU utilization over autoregressive decoding by generating multiple tokens per forward pass.<n>Their KV cache still grows linearly with context, limiting throughput at long contexts.<n>We present HERALD, a KV offloading system for block dLLMs.
• Score: 31.594043836825403
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion LLMs (dLLMs) improve GPU utilization over autoregressive decoding by generating multiple tokens per forward pass, but their KV cache still grows linearly with context, limiting throughput at long contexts. KV cache offloading to host DRAM alleviates this memory pressure, but the limited PCIe bandwidth necessitates recalling only a sparse subset of KV entries. In block dLLMs, the relevant KV entries remain consistent across denoising steps within a block, enabling high-accuracy selection by identifying the top-k entries once and reusing them throughout all denoising steps. This property appears attractive for offloading as it amortizes the selection overhead across the entire block, but it requires exact attention over the full KV cache, which is too expensive under offloading. We present HERALD, a KV offloading system for block dLLMs that resolves this through two opportunities that reduce the required selection compute by a factor of the block size and enable selection to be overlapped with denoising. Across three block dLLMs and five long context tasks, HERALD achieves near-lossless accuracy at 5-10% KV budget and up to 1.59x lower per block latency and 2.47x higher throughput over GPU-only inference, with speedups growing with context length.

Related papers

• SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference [37.37437232987781]
  SeKV is a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans.<n>We show that SeKV improves over the strongest semantic compression baseline by 5.9% on average.<n>It reduces GPU memory by 53.3% versus full KV caching at 128K context.
  arXiv  Detail & Related papers  (2026-06-30T05:18:02Z)
• KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference [23.683284557561112]
  We present KVDrive, a holistic multi-tier KV cache management system spanning GPU memory, host DRAM, and SSD.<n> KVDrive tackles the problem from a systems perspective - jointly orchestrating cache placement, pipeline scheduling, and cross-tier coordination.<n>The system achieves up to 1.74x higher throughput compared to state-of-the-art works while preserving accuracy.
  arXiv  Detail & Related papers  (2026-05-18T08:54:16Z)
• Prefix-Adaptive Block Diffusion for Efficient Document Recognition [52.15352911463151]
  Block Diffusion Models (BDMs) support parallel generation, flexible-length output, and KV caching, making them promising for efficient document parsing.<n>We propose the Prefix- Block Diffusion Model (PA-BDM) which replaces intra-block bidirectional denoising with causal denoising from prefix to suffix.<n> Experiments show that the 3B PA-BDM achieves higher recognition scores on several benchmarks and improves inference throughput by 71.6% over the 2.5B MinerU-Diffusion.
  arXiv  Detail & Related papers  (2026-05-16T07:50:13Z)
• CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection [13.937483869660648]
  Existing sparse attention methods do not translate efficiently to chunked prefill.<n>We propose CompactAttention, a chunked-prefill attention mechanism based on Block-Union KV Selection.<n>It delivers up to 2.72$times$ attention speedup at 128K context length under chunked prefill.
  arXiv  Detail & Related papers  (2026-05-16T06:47:41Z)
• Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving [18.681651346752766]
  Dynamic sparse attention promises relief by accessing only a small, query-dependent subset of the KV state per decoding step.<n>In practice, however, these algorithmic savings rarely translate into end-to-end system-level gains.<n>We present SPIN, a sparse-attention-aware inference framework that co-designs the execution pipeline with hierarchical KV storage.
  arXiv  Detail & Related papers  (2026-04-29T16:02:00Z)
• LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models [49.93891888238178]
  Block-wise diffusion language models (DLMs) generate multiple tokens in any order, offering a promising alternative to the autoregressive decoding pipeline.<n>Naive sparse attention fails on DLMs due to a KV Inflation problem, where different queries select different prefix positions.<n>We propose LOSA (Locality-aware Sparse Attention), which reuses cached prefix-attention results for stable tokens and applies sparse attention only to active tokens.
  arXiv  Detail & Related papers  (2026-04-13T20:53:51Z)
• MAGE: All-[MASK] Block Already Knows Where to Look in Diffusion LLM [9.69241599043101]
  Block diffusion LLMs are emerging as a promising next paradigm for language generation, but their use of KV caching makes memory access a dominant bottleneck in long-context settings.<n>This work identifies a key opportunity unique to block diffusion: attention at the first All-[MASK] denoising step reliably predicts important KV entries and budget requirements.<n>MAGE achieves near-lossless accuracy with a fraction of the KV budget while delivering up to 3-4x end-to-end speedup.<n>A lightweight fine-tuning strategy further strengthens [MASK]-guided patterns with minimal cost, requiring only a few hours of training on
  arXiv  Detail & Related papers  (2026-02-15T16:07:51Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• NOSA: Native and Offloadable Sparse Attention [27.551376861663556]
  We present NOSA, a trainable sparse attention framework designed to support KV cache offloading.<n>We show that NOSA preserves near-lossless performance while achieving up to a 2.3x improvement in decoding throughput.
  arXiv  Detail & Related papers  (2025-10-15T14:33:16Z)
• Fast-dLLM v2: Efficient Block-Diffusion LLM [64.38006546510337]
  Fast-dLLM v2 is a block diffusion language model that adapts pretrained AR models into dLLMs for parallel text generation.<n>This represents a 500x reduction in training data compared to full-attention diffusion LLMs such as Dream (580B tokens)
  arXiv  Detail & Related papers  (2025-09-30T14:40:18Z)
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction [72.27673320976933]
  Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding.<n>Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage.<n>We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention.
  arXiv  Detail & Related papers  (2025-08-04T16:14:03Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.