Attention-Based Sampler for Diffusion Language Models

• URL: http://arxiv.org/abs/2604.08564v1
• Date: Wed, 18 Mar 2026 07:49:13 GMT
• Title: Attention-Based Sampler for Diffusion Language Models
• Abstract summary: We propose a new training-free decoding algorithm, termed Attn-Sampler.<n>We show that optimal sequence likelihood can be approximately achieved by decoding tokens in descending order of their attention column sums.
• Score: 16.008464195041984
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Auto-regressive models (ARMs) have established a dominant paradigm in language modeling. However, their strictly sequential decoding paradigm imposes fundamental constraints on both inference efficiency and modeling flexibility. To address these limitations, diffusion-based large language models (dLLMs) have been proposed, offering the potential for parallel decoding and flexible language modeling. Despite these advantages, current dLLMs decoding strategies rely primarily on token level information, which fails to account for global sequence structure and often yields suboptimal results. In this paper, we study the decoding order selection problem from the perspective of log-likelihood maximization. We theoretically demonstrate that optimal sequence likelihood can be approximately achieved by decoding tokens in descending order of their attention matrix column sums. This finding provides a principled justification for attention-guided decoding and offers a theoretically grounded alternative to greedy search. We instantiate this theoretical insight in a new training-free decoding algorithm, termed Attn-Sampler, and further propose a block attention approximation and dynamic attention thresholding for practical acceleration. Extensive experiments across multiple benchmarks validate the effectiveness of our proposed method, demonstrating that it achieves superior generation quality while enhancing the decoding parallelism.

Related papers

• Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models [129.8880969196575]
  We present an empirical analysis of attention in Diffusion large language models (dLLMs)<n>We show that tokens attending more strongly to unmasked context exhibit greater generation stability and play a critical role in reasoning.<n>Motivated by these findings, we propose AGDO, an attention-guided denoising and optimization framework.
  arXiv  Detail & Related papers  (2026-06-10T16:14:23Z)
• Confidence-Based Decoding is Provably Efficient for Diffusion Language Models [8.527157686215117]
  Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models for language modeling.<n>We develop the first theoretical analysis framework for confidence-based decoding in DLMs.
  arXiv  Detail & Related papers  (2026-03-23T17:43:21Z)
• Decoding Large Language Diffusion Models with Foreseeing Movement [36.326379474672564]
  Large Language Diffusion Models (LLDMs) benefit from a flexible decoding mechanism that enables parallelized inference and controllable generations over autoregressive models.<n>Existing methods, however, focus mainly on local effects while overlooking long-term impacts.<n>We propose the Foreseeing Decoding Method (FDM), a novel approach that integrates both local and global considerations to unlock the full potential.
  arXiv  Detail & Related papers  (2025-12-03T14:39:41Z)
• Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective [85.06838178922791]
  Reinforcement Learning (RL) has proven highly effective for autoregressive language models.<n>But adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges.<n>We propose a principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy.
  arXiv  Detail & Related papers  (2025-12-03T13:05:32Z)
• Language Ranker: A Lightweight Ranking framework for LLM Decoding [70.01564145836129]
  This paper conceptualizes the decoding process as analogous to the ranking stage in recommendation pipelines.<n>Motivated by this insight, we propose Language Ranker, a novel framework that introduces a lightweight module to rerank candidate responses.<n> Experiments show that Language Ranker achieves performance comparable to large-scale reward models, while requiring only 0.5M additional parameters.
  arXiv  Detail & Related papers  (2025-10-23T17:56:46Z)
• Self-Augmented Visual Contrastive Decoding [13.078702859025482]
  Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal capabilities.<n>LVLMs inherit the tendency to hallucinate from their underlying language models.<n>This study introduces a novel training-free decoding strategy that addresses these limitations.
  arXiv  Detail & Related papers  (2025-10-15T09:03:34Z)
• Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models [8.407364705777587]
  We introduce Free Draft-and-Verification (FreeDave), a novel fast decoding algorithm tailored forDLLMs.<n>FreeDave is proven to boost the inference throughput up to $3.78times$ without performance degradation.
  arXiv  Detail & Related papers  (2025-09-30T21:28:04Z)
• A Theoretical Perspective for Speculative Decoding Algorithm [60.79447486066416]
  One effective way to accelerate inference is emphSpeculative Decoding, which employs a small model to sample a sequence of draft tokens and a large model to validate. This paper tackles this gap by conceptualizing the decoding problem via markov chain abstraction and studying the key properties, emphoutput quality and inference acceleration, from a theoretical perspective.
  arXiv  Detail & Related papers  (2024-10-30T01:53:04Z)
• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• Amortizing intractable inference in large language models [56.92471123778389]
  We use amortized Bayesian inference to sample from intractable posterior distributions. We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem.
  arXiv  Detail & Related papers  (2023-10-06T16:36:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.