PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models

• URL: http://arxiv.org/abs/2605.20813v1
• Date: Wed, 20 May 2026 07:06:54 GMT
• Title: PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models
• Abstract summary: Inference in diffusion large language models (dLLMs) is computationally expensive.<n>Recent sparse attention methods for dLLMs mitigate this cost via block-sparse computation.<n>We introduce PulseCol, a periodically refreshed column-sparse attention method.
• Score: 61.39458217121294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inference in diffusion large language models (dLLMs) is computationally expensive, as full self-attention must be repeatedly executed at each step of the denoising process without KV cache. Recent sparse attention methods for dLLMs mitigate this cost via block-sparse computation, which is applied only in later iterations when model performance is less sensitive to coarse-grained sparse approximation, but yields limited improvements in computational efficiency and acceleration. This motivates a finer-grained sparsification strategy that can be applied from earlier iterations and leverages reusable sparsity patterns, enabling further efficiency gains. In this work, we introduce PulseCol, a periodically refreshed column-sparse attention method for accelerating diffusion language models. PulseCol replaces coarse block-level sparsity with a finer-grained column-sparse structure, allowing important attention interactions to be retained more precisely while exposing greater sparsity. Built on this column-level formulation, PulseCol further identifies sparse patterns at the early denoising step and reuses them across subsequent iterations, refreshing them only at a small number of intermediate steps to track the evolution of sparse attention patterns during denoising. Experiments show that PulseCol achieves higher sparsity and greater practical speedup than prior sparse attention methods for dLLMs, while maintaining model quality. Enabled by optimized GPU kernels for column-sparse attention, PulseCol delivers up to 1.95$\times$ end-to-end speedup over FlashAttention across several context lengths.

Related papers

• Evolving Cache Schedules for Fast Diffusion Policy Inference [6.879031540914794]
  Diffusion policies achieve strong visuomotor control by iteratively denoising action chunks, but repeated denoising makes real-time deployment computationally demanding.<n>We introduce Evolving Cache Schedules (EVO), a training-free acceleration framework that globally schedules cache refreshes via evolutionary search.<n>EVO preserves near-full performance while substantially reducing computation, achieving up to 8.05x action-generation speedup and reducing FLOPs from 15.77G to as low as 1.96G.
  arXiv  Detail & Related papers  (2026-07-22T15:40:11Z)
• Flash-BoN: Instant Drafts for Inference-Time Scaling in Diffusion Models [91.19049123903137]
  Inference-time scaling for text-to-image generation has progressed from simple Best-of-$N$ sampling to guided search methods.<n>These approaches focus on when and how often to verify during denoising but largely treat the cost of generation itself as fixed.<n>We show that under wall-clock evaluation, simple BoN already matches or outperforms several guided search techniques.
  arXiv  Detail & Related papers  (2026-07-05T19:02:32Z)
• Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention [58.484607892210015]
  Diffusion Language Models (DLMs) enable globally coherent, bidirectional, and controllable text generation.<n>Many existing block-sparse attention methods select blocks by fixed sampling patterns over the high-resolution attention space.<n>We propose the Block Approximate Sparse Attention framework (BA-Att) with block-wise pre-downsampled operation.
  arXiv  Detail & Related papers  (2026-05-19T12:01:13Z)
• Timestep-Aware Block Masking for Efficient Diffusion Model Inference [48.764748095994996]
  Diffusion Probabilistic Models (DPMs) have achieved great success in image generation but suffer from high inference latency.<n>We propose a novel framework to optimize the computational graph of pre-trained DPMs on a per-timestep basis.
  arXiv  Detail & Related papers  (2026-03-20T13:35:03Z)
• FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling [43.057651076580264]
  FlashPrefill is a framework enabling ultra-fast prefilling via instantaneous pattern discovery and thresholding.<n>FlashPrefill achieves a substantial leap in efficiency, delivering an unprecedented 27.78x speedup on 256K sequences.<n>Unlike existing methods that incur efficiency degradation on shorter contexts, FlashPrefill maintains a 1.71x speedup even at a 4K context length.
  arXiv  Detail & Related papers  (2026-03-06T12:12:46Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• LiteAttention: A Temporal Sparse Attention for Diffusion Transformers [1.3471268811218626]
  LiteAttention exploits temporal coherence to enable evolutionary computation skips across the denoising sequence.<n>We implement a highly optimized LiteAttention kernel on top of FlashAttention and demonstrate substantial speedups on production video diffusion models.
  arXiv  Detail & Related papers  (2025-11-14T08:26:55Z)
• OmniSAT: Compact Action Token, Faster Auto Regression [70.70037017501357]
  We introduce an Omni Swift Action Tokenizer, which learns a compact, transferable action representation.<n>The resulting discrete tokenization shortens the training sequence by 6.8$times$, and lowers the target entropy.
  arXiv  Detail & Related papers  (2025-10-08T03:55:24Z)
• SparseD: Sparse Attention for Diffusion Language Models [98.05780626106555]
  diffusion language models (DLMs) offer a promising alternative to autoregressive models (ARs)<n>Existing open-source DLMs suffer from high inference latency.<n>We propose SparseD, a novel sparse attention method for DLMs.
  arXiv  Detail & Related papers  (2025-09-28T18:10:10Z)
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction [72.27673320976933]
  Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding.<n>Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage.<n>We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention.
  arXiv  Detail & Related papers  (2025-08-04T16:14:03Z)
• FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion [22.207275433870937]
  Diffusion language models offer parallel token generation and inherent bidirectionality.<n>State-of-the-art diffusion models (e.g., Dream 7B, LLaDA 8B) suffer from slow inference.<n>We introduce Guided Diffusion, a training-free method that uses a lightweight pretrained autoregressive model to supervise token unmasking.
  arXiv  Detail & Related papers  (2025-05-27T17:39:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.