Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching

• URL: http://arxiv.org/abs/2601.20332v2
• Date: Sun, 01 Feb 2026 13:11:03 GMT
• Title: Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching
• Abstract summary: We present a window-based token pruning and caching method for inference.<n>Experiments on LLaDA and Dream show that, under matched compute budgets, our method achieves up to $99times$ inference speedup.
• Score: 10.315266731366123
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion language models (DLMs) generate text through iterative denoising, but inference requires full-sequence attention at every iteration, resulting in substantial redundant computation on masked tokens. Block-wise diffusion can reduce this cost, yet it typically relies on retraining and constrained update orders, limiting its direct applicability to pretrained DLMs. Our token-level analysis reveals pronounced structural locality in DLM inference. Decoding is driven by a small set of prefix-localized active tokens; the influence of distant undecoded context diminishes rapidly, and decoded tokens exhibit stage-wise temporal stability, enabling reuse of intermediate representations except for a brief post-decode transient. Motivated by these observations, we propose \textbf{\placeholder}\footnote{The source code is available at https://github.com/vhicrgit/Window-Diffusion.}, a window-based token pruning and caching method for inference. We maintain a local computation window that slides rightward as denoising progresses, and partition undecoded tokens into: (i) \textit{active tokens} that are computed online, (ii) \textit{buffer tokens} whose KV states are cached and periodically refreshed, and (iii) \textit{far-field tokens} that are pruned outside the window. Computation is restricted to active and buffer tokens within the window, while far-field tokens are omitted at each stage. Experiments on LLaDA and Dream show that, under matched compute budgets, our method achieves up to $99\times$ inference speedup while largely preserving generation performance.

Related papers

• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• SAID: Accelerating Diffusion-Based Language Models via Scaffold-Aware Iterative Decoding [11.233923872369202]
  Diffusion large language models (DLLMs) enable non-autoregressive computation generation by iteratively denoising corrupted tokens with bidirectional context.<n>We propose SAID, a Scaffold-Aware Iterative Decoding framework that acceleratesDLLMs by reallocating across tokens.<n>Experiments on LLaDA-8B and LLaDA 1.5 across math, coding, and knowledge benchmarks show that SAID significantly accelerates inference with a maximum speedup of 9.1x while maintaining competitive performance.
  arXiv  Detail & Related papers  (2026-06-03T14:56:42Z)
• SimSD: Simple Speculative Decoding in Diffusion Language Models [61.33773959352141]
  Diffusion large language models (dLLMs) offer faster inference through parallel or blockwise decoding.<n>Their masked language modeling formulation remains incompatible with standard token-level speculative decoding.<n>We propose a speculative decoding algorithm for dLLMs, named SimSD, that equips dLLMs with temporally valid token-level contexts.<n>Our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
  arXiv  Detail & Related papers  (2026-06-01T17:46:46Z)
• DiffuMask: Diffusion Language Model for Token-level Prompt Pruning [43.97675296962604]
  In-Context Learning and Chain-of-Thought prompting improve reasoning in large language models (LLMs)<n>Prompt compression based on pruning offers a practical solution, yet existing methods rely on sequential token removal which is computationally intensive.<n>We present DiffuMask, a diffusion-based framework integrating hierarchical shot-level and token-level pruning signals.
  arXiv  Detail & Related papers  (2026-04-08T03:08:18Z)
• DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention [2.7422645382944935]
  We present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only salient tokens.<n>Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput.
  arXiv  Detail & Related papers  (2026-03-09T07:02:01Z)
• Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models [61.11154533305096]
  Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens.<n>We propose a new perspective that elaborates token textbfAnchors within intra-frame and inter-frame contexts.<n>Our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs.
  arXiv  Detail & Related papers  (2026-03-02T03:06:40Z)
• Residual Context Diffusion Language Models [90.07635240595926]
  Residual Context Diffusion (RCD) is a module that converts discarded token representations into contextual residuals and injects them back for the next denoising step.<n>RCD consistently improves frontier dLLMs by 5-10 points in accuracy with minimal extra computation overhead.
  arXiv  Detail & Related papers  (2026-01-30T13:16:32Z)
• Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding [36.74241893088594]
  Diffusion Large Language Models (dLLMs) offer a compelling paradigm for natural language generation.<n>Recent works have accelerated inference via KV cache reuse or decoding, but overlook the intrinsic inefficiencies within the block-wise diffusion process.<n>We propose Streaming-dLLM, a training-free framework that streamlines inference across both spatial and temporal dimensions.
  arXiv  Detail & Related papers  (2026-01-25T17:36:04Z)
• WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference [44.87788417755154]
  We propose WeDLM, a diffusion decoding framework built entirely on standard causal attention.<n>We show that WeDLM preserves the quality of strong AR backbones while delivering substantial speedups.
  arXiv  Detail & Related papers  (2025-12-28T01:25:48Z)
• Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs [55.827877498548965]
  We propose a lightweight training framework that learns a single prompt-specific Behavior-Equivalent token ([BE])<n>The framework first trains [BE] to encode the natural-language content of the original system prompt via reconstruction, and then distills the prompt's downstream behavior into this single token.<n> Empirical evaluations on three datasets show that a single [BE] token achieves up to a 3000x reduction in prompt length, while retaining about 98% of the downstream performance of the original system prompts.
  arXiv  Detail & Related papers  (2025-11-28T15:22:52Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Models [40.902681492117786]
  RemeDi is a mask-based DLM that predicts token distributions and per-token confidence scores at each step.<n>We train a remask-aware pipeline to train this ability, including supervised fine-tuning which teaches the model to detect and remask incorrect tokens.<n>Experiments show that RemeDi achieves the state-of-the-art results among open-source DLMs on multiple datasets.
  arXiv  Detail & Related papers  (2025-09-28T05:39:49Z)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning [23.58934174168992]
  Autoregressive (AR) language models generate text one token at a time, which limits their inference speed.<n>We propose Convolutional decoding (Conv), a normalization-based method that narrows the decoding window without hard segmentation.<n>We also introduce Rejecting Rule-based Fine-Tuning (R2FT), a post-hoc training scheme that better aligns tokens at positions far from context.
  arXiv  Detail & Related papers  (2025-09-18T17:48:21Z)
• SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning [3.502168555273189]
  SlimInfer aims to accelerate inference by directly pruning less critical prompt tokens during the forward pass.<n>We show that SlimInfer can achieve up to $mathbf2.53times$ time-to-first-token (TTFT) speedup and $mathbf1.88times$ end-to-end latency reduction for LLaMA3.1-8B-Instruct.
  arXiv  Detail & Related papers  (2025-08-08T16:42:38Z)
• VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models [35.38573641029626]
  We introduce the novel task of Extreme Short Token Reduction, which aims to represent entire videos using a minimal set of discrete tokens.<n>On the Extreme Short Token Reduction task, our VQToken compresses sequences to just 0.07 percent of their original length while incurring only a 0.66 percent drop in accuracy on the NextQA-MC benchmark.
  arXiv  Detail & Related papers  (2025-03-21T09:46:31Z)
• RefreshKV: Updating Small KV Cache During Long-form Generation [54.00118604124301]
  We propose a new inference method, RefreshKV, that flexibly alternates between full context attention and attention over a subset of input tokens during generation.<n>Applying our method to off-the-shelf LLMs achieves comparable speedup to eviction-based methods while improving performance for various long-form generation tasks.
  arXiv  Detail & Related papers  (2024-11-08T18:57:07Z)
• FIRP: Faster LLM inference via future intermediate representation prediction [54.897493351694195]
  FIRP generates multiple tokens instead of one at each decoding step. We conduct extensive experiments, showing a speedup ratio of 1.9x-3x in several models and datasets.
  arXiv  Detail & Related papers  (2024-10-27T15:53:49Z)
• Training-Free Exponential Context Extension via Cascading KV Cache [49.608367376911694]
  We introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens.<n>Our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens.
  arXiv  Detail & Related papers  (2024-06-24T03:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.