Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

• URL: http://arxiv.org/abs/2609.11998v1
• Date: Wed, 09 Sep 2026 23:37:47 GMT
• Title: Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
• Abstract summary: Block diffusion restores caching by decoding block-by-block.<n>We study this recipe at scale, pretraining three 3B block-diffusion denoisers on 300B tokens.
• Score: 18.318746660954663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion language models decode tokens in parallel, but their bidirectional denoiser rules out the naive key--value (KV) cache behind fast autoregressive inference. Block diffusion restores caching by decoding block-by-block, and the block caches deployed on it so far are tied to attention: O(L)in memory and, if used as training-free retrofits, only an approximation of the model's computation. Both constraints can be overcome: sequence mixers that summarize finalized blocks into a reusable state support block caching, and the corresponding block-causal training objective makes the cache exact. We study this recipe at scale, pretraining three 3B block-diffusion denoisers (attention, mamba, and hybrid) on 300B tokens under one single-frontier objective and decoding all three through a single cached interface. Only the state-space cache is O(1) in sequence length: its memory and per-step latency stay constant at any context length, while an attention cache remains O(L). At 256k tokens (where attention has grown to 82GB and 29 ms/step), the Mamba cache delivers 4.3x lower latency, 11x less memory, and 2.6x higher single-stream throughput; and because that footprint is constant it scales with batch as well, reaching 14x the aggregate throughput, where attention cannot run beyond a single stream. The same linear-state bias lets the Mamba and hybrid backbones keep retrieving out to 8-16x their training length, whereas attention's retrieval collapses at 2x, at no measured quality cost.

Related papers

• LaCache: Exact Caching and Precision-Adaptive Inference for Diffusion Large Language Models [9.146969118551167]
  DLLMs enable parallel generation via Semi-Autoregressive (SAR) decoding in text generation.<n>Current methods suffer from severe operator-level redundancy.<n>We propose LaCache, a training-free acceleration framework.
  arXiv  Detail & Related papers  (2026-07-16T10:49:15Z)
• Training Hybrid Block Diffusion Language Models with Partial Bidirectionality [11.618306403714692]
  Long-context generation is one of the central challenges for large language models.<n>We propose a BDLM Mamba--attention hybrid that addresses this challenge by restricting the reverse Mamba scan to the active denoising block.<n>For long-context inference, BDLM Mamba-H reaches 19.7x the throughput of full-sequence DiffuMamba-H at 65K tokens and 3.7x the throughput of BDLM attention at 262K.
  arXiv  Detail & Related papers  (2026-07-02T22:37:43Z)
• Prefix-Adaptive Block Diffusion for Efficient Document Recognition [52.15352911463151]
  Block Diffusion Models (BDMs) support parallel generation, flexible-length output, and KV caching, making them promising for efficient document parsing.<n>We propose the Prefix- Block Diffusion Model (PA-BDM) which replaces intra-block bidirectional denoising with causal denoising from prefix to suffix.<n> Experiments show that the 3B PA-BDM achieves higher recognition scores on several benchmarks and improves inference throughput by 71.6% over the 2.5B MinerU-Diffusion.
  arXiv  Detail & Related papers  (2026-05-16T07:50:13Z)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion [61.57938553036056]
  ARL2 is a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state.<n>We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame recurrent linear branch that maintains a fixed-size state for streaming context.<n>With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-15T19:33:45Z)
• RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache [28.54642982960947]
  Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memory size and bandwidth.<n>Existing methods reduce the cache by either eviction or quantization, but typically treat the two in isolation.<n>In this paper, we cast KV cache compression as a rate-distortion problem, under which eviction and quantization are two end-points of the same bit allocation scheme.
  arXiv  Detail & Related papers  (2026-05-08T15:15:06Z)
• Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts [68.79341332280062]
  Training Large Language Models (LLMs) on long contexts is severely constrained by prohibitive GPU memory overhead, not training time.<n>We introduce OOMB, a highly memory-efficient training system that directly confronts this barrier.<n>Our approach employs a chunk-recurrent training framework with on-the-fly activation recomputation, which maintains a constant activation memory footprint.
  arXiv  Detail & Related papers  (2026-02-02T13:52:40Z)
• No Cache Left Idle: Accelerating diffusion model via Extreme-slimming Caching [17.396336005757025]
  We present X-Slim (e-Xtreme-Slimming Caching), a training-free, cache-based accelerator.<n>It is the first unified framework to exploit cacheable redundancy across timesteps, structure (blocks), and space (tokens)<n>It reduces latency by up to 4.97x and 3.52x with minimal perceptual loss.
  arXiv  Detail & Related papers  (2025-12-14T09:02:18Z)
• VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization [23.781285860723248]
  Key-Value ( KV) cache introduces memory overhead during large language model (LLM) inference.<n>We propose VecInfer, a novel VQ method for aggressive KV cache compression while enabling efficient inference.<n>VecInfer consistently outperforms existing quantization baselines across both long-context understanding and mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2025-10-07T17:35:28Z)
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction [72.27673320976933]
  Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding.<n>Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage.<n>We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention.
  arXiv  Detail & Related papers  (2025-08-04T16:14:03Z)
• Training-Free Exponential Context Extension via Cascading KV Cache [49.608367376911694]
  We introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens.<n>Our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens.
  arXiv  Detail & Related papers  (2024-06-24T03:59:17Z)
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache [67.9776980972508]
  We develop a tuning-free 2bit KV cache quantization algorithm named KIVI. KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using $mathbf2.6times$ less peak memory.
  arXiv  Detail & Related papers  (2024-02-05T06:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.