Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models

• URL: http://arxiv.org/abs/2605.29607v1
• Date: Thu, 28 May 2026 08:42:39 GMT
• Title: Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models
• Abstract summary: Masked diffusion language models (MDLMs) enable parallel decoding by predicting all masked positions at each denoising step.<n>We revisit this granularity and observe that reliable predictions often emerge as contiguous high-confidence spans.<n>We use self-attention maps to estimate inter-cluster dependencies, enabling conflict-aware selection of mutually compatible CICs for parallel commitment.
• Score: 13.325071163425621
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked diffusion language models (MDLMs) enable parallel decoding by predicting all masked positions at each denoising step, yet existing training-free samplers usually decide which positions to commit at token-level granularity. We revisit this granularity and observe that reliable predictions often emerge as contiguous high-confidence spans, suggesting that the unit of parallel commitment can be larger than a single token. We first group adjacent high-confidence candidates into confidence-induced clusters (CICs) as span-level update units. We then use self-attention maps from the same forward pass to estimate inter-cluster dependencies, enabling conflict-aware selection of mutually compatible CICs for parallel commitment. This yields CLAD (Cluster-Level Attention-Guided Decoding), a training-free cluster-level decoder for MDLMs. Experiments on LLaDA and Dream model families across four reasoning and code-generation benchmarks show that CLAD achieves 1.77x--8.47x speedups over Vanilla decoding while maintaining broadly comparable task accuracy in most settings.

Related papers

• Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling [11.823547220829083]
  Diffusion large language models (dLLMs) offer a promising parallel decoding paradigm as an alternative to autoregressive generation.<n>We propose an "early-bird (EB)" decoding framework, motivated by the observation that tokens with similarly low entropy tend to cluster.<n>EB-Decode integrates two key enablers: (1) a learnable network that adaptively groups tokens with similar uncertainty into variable-length blocks, rather than relying on fixed block sizes; and (2) a position-aware sampler that learns to unmask tokens in parallel using fewer decoding steps within predicted variable-length blocks.
  arXiv  Detail & Related papers  (2026-09-15T00:12:37Z)
• Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs [7.894740505622926]
  Diffusion multimodal large language models (dMLLMs) produce long-form outputs marred by semantic drift and repetition.<n>We propose ours (textbfContext-textbfAware textbfCluster textbfDecoding), a training-free decoding method that scores each masked position by a multiplicative composite of softmax confidence and neighbor proximity.
  arXiv  Detail & Related papers  (2026-08-23T11:16:04Z)
• Ripple-Pivot Search: Active Parallel Decoding for Diffusion Large Language Models [66.32132912853372]
  Diffusion Large Language Models (dLLMs) have emerged as a competitive alternative to autoregressive language models.<n>We propose Ripple-Pivot Search (RPS), a training-free decoding method that seeks mid-entropy positions as promising candidate pivots.<n>RPS achieves 4-10$times$ wall-clock speedup over the standard decoder while preserving generation quality, and improves accuracy over the previous lookahead baseline by up to 5.49%.
  arXiv  Detail & Related papers  (2026-08-12T07:32:29Z)
• Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models [59.51249894128724]
  Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration.<n>We propose ADAS, a training-free reranking rule for parallel masked diffusion decoding.
  arXiv  Detail & Related papers  (2026-06-09T13:17:27Z)
• FLARE: Diffusion for Hybrid Language Model [72.60770374799634]
  FLARE is a systematic conversion framework for hybrid-attention large language models.<n>It combines a token-equal AR-and-diffusion objective, hardware-aware kernels, and unified inference, enabling one checkpoint to support both AR-style verified decoding and diffusion-style parallel denoising.<n>Our results suggest practical dLLMs are limited not only by decoding algorithms, but also by transfer data quality and the training inefficiency of current block-diffusion objectives.
  arXiv  Detail & Related papers  (2026-06-01T06:58:15Z)
• LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection [35.2079721755684]
  We introduce LEAP (Lookahead Early-Convergence Token Detection for Accelerated Parallel Decoding)<n>LEAP is a training-free, plug-and-play method that leverages future context filtering and multi-sequence superposition to detect early-converging tokens.<n>On the GSM8K dataset, combining LEAP with dParallel accelerates decoding to 7.2 tokens per step while preserving model precision.
  arXiv  Detail & Related papers  (2026-05-09T03:26:23Z)
• CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning [79.07656918674684]
  CoME-VL is a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder.<n>Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin.
  arXiv  Detail & Related papers  (2026-04-03T17:59:51Z)
• Dependency-Guided Parallel Decoding in Discrete Diffusion Language Models [21.699371484195865]
  We propose DEMASK, a lightweight dependency predictor that attaches to the final hidden states of a dLLM.<n> Empirically, DEMASK achieves 1.7-2.2$times$ speedup on Dream-7B while matching or improving accuracy compared to confidence-based and KL-based baselines.
  arXiv  Detail & Related papers  (2026-04-02T22:21:24Z)
• DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention [2.7422645382944935]
  We present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only salient tokens.<n>Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput.
  arXiv  Detail & Related papers  (2026-03-09T07:02:01Z)
• Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding [6.755667885643806]
  Diffusion-based large language models (dLLMs) have shown promising performance across various reasoning tasks.<n>We introduce an adaptive parallel decoding approach, namely DiCo, which features a three-phase divide-and-conquer paradigm.<n>Extensive experiments demonstrate that DiCo can achieve significant inference speedups while maintaining competitive generation quality.
  arXiv  Detail & Related papers  (2026-02-27T08:36:06Z)
• Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models [58.946955321428845]
  This work presents self-rewarding sequential Monte Carlo (SMC)<n>Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy.<n>We introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights.
  arXiv  Detail & Related papers  (2026-02-02T09:21:45Z)
• CDLM: Consistency Diffusion Language Models For Faster Sampling [54.886467592798]
  Diffusion Language Models (DLMs) offer a promising parallel generation paradigm but suffer from slow inference.<n>We introduce CDLM, a training-based acceleration method that simultaneously tackles both bottlenecks.<n>Experiments show CDLM achieves 3.6x-14.5x lower latency while maintaining competitive accuracy on math and coding tasks.
  arXiv  Detail & Related papers  (2025-11-24T16:21:25Z)
• Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing [4.707859580472452]
  Masked diffusion models (MDMs) offer a compelling alternative to autoregressive models (ARMs) for discrete text generation.<n>They enable parallel token sampling, rather than sequential, left-to-right generation.<n>We present PUNT, a model-agnostic sampler that reconciles this trade-off.
  arXiv  Detail & Related papers  (2025-10-24T18:41:26Z)
• Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech [51.14752758616364]
  Speech-based depression detection (SDD) is a promising, non-invasive alternative to traditional clinical assessments.<n>We propose HAREN-CTC, a novel architecture that integrates multi-layer SSL features using cross-attention within a multitask learning framework.<n>The model achieves state-of-the-art macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA, outperforming prior methods across both evaluation scenarios.
  arXiv  Detail & Related papers  (2025-10-05T09:32:12Z)
• Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency [57.961869351897384]
  We propose a framework based on cross-modal semantic consistency for efficient image clustering.<n>Our framework first builds a strong foundation via Cross-Modal Semantic Consistency.<n>In the first stage, we train lightweight clustering heads to align with the rich semantics of the pre-trained model.<n>In the second stage, we introduce a Self-Enhanced fine-tuning strategy.
  arXiv  Detail & Related papers  (2025-08-02T08:12:57Z)
• Towards Learnable Anchor for Deep Multi-View Clustering [49.767879678193005]
  In this paper, we propose the Deep Multi-view Anchor Clustering (DMAC) model that performs clustering in linear time.<n>With the optimal anchors, the full sample graph is calculated to derive a discriminative embedding for clustering.<n>Experiments on several datasets demonstrate superior performance and efficiency of DMAC compared to state-of-the-art competitors.
  arXiv  Detail & Related papers  (2025-03-16T09:38:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.