On the Reasoning Abilities of Masked Diffusion Language Models

• URL: http://arxiv.org/abs/2510.13117v1
• Date: Wed, 15 Oct 2025 03:29:26 GMT
• Title: On the Reasoning Abilities of Masked Diffusion Language Models
• Abstract summary: Masked diffusion models (MDMs) for text offer a compelling alternative to traditional autoregressive language models.<n>We characterize what types of reasoning problems MDMs can provably solve and how efficiently.
• Score: 34.937359931224265
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked diffusion models (MDMs) for text offer a compelling alternative to traditional autoregressive language models. Parallel generation makes them efficient, but their computational capabilities and the limitations inherent to their parallelism remain largely unexplored. To this end, we characterize what types of reasoning problems MDMs can provably solve and how efficiently. We do this by connecting MDMs to the well-understood reasoning frameworks of chain of thought (CoT) and padded looped transformers (PLTs) in the finite-precision log-width setting: We show that MDMs and polynomially-padded PLTs are, in fact, equivalent in this setting, and that MDMs can solve all problems that CoT-augmented transformers can. Moreover, we showcase classes of problems (including regular languages) for which MDMs are inherently more efficient than CoT transformers, where parallel generation allows for substantially faster reasoning.

Related papers

• Semantic Steering for Controllable Generation: Tuning-Free Concept Erasure in Multimodal Diffusion Transformers [22.773634194858037]
  Multimodal Diffusion Transformers (MM-DiTs) have demonstrated remarkable text-to-image generation performance.<n>Their powerful generative capabilities also raise significant safety concerns, as they may generate sensitive or inappropriate content.<n>We propose a tuning-free approach to erase concepts by directly manipulating the model's internal representations.
  arXiv  Detail & Related papers  (2026-08-13T05:05:45Z)
• Multi-Mask Diffusion Language Models for Few-Step Generation [67.14463385328135]
  Masked diffusion models (MDMs) are a promising family of language generators.<n>Recent few-step alternatives based on uniform-state diffusion avoid this degeneracy.<n>We propose a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation.
  arXiv  Detail & Related papers  (2026-07-22T02:35:40Z)
• Masked Language Flow Models [10.204151640924454]
  Flow Language Models (FLMs) sidestep this limitation by learning a continuous flow that transports noise toward clean sequences.<n>We introduce Masked Language Flow Models (MLFMs) which incorporate masking into FLMs using a continuous interpolant to bridge partially masked and clean sequences.<n>We find, for the first time, that flow-based language models can be scaled to solve downstream reasoning and instruction-following tasks.
  arXiv  Detail & Related papers  (2026-06-26T00:16:40Z)
• Posterior Refinement: Fast Language Generation via Any-Order Flow Maps [36.18484491074518]
  Flow Map Language Models (FMLMs) circumvent this bottleneck via joint sequence transport for excellent few-step generation.<n>We introduce FMLM+, a framework that bridges this gap by equipping FMLM with masking-style noise schedules.<n>We leverage this to introduce Posterior Refinement, a novel inference-time refinement strategy that enables the model to adaptively self-correct its outputs.
  arXiv  Detail & Related papers  (2026-06-23T16:33:13Z)
• Multi-Turn Reflective Masking Elicits Reasoning in Mask Diffusion Models [40.620031617929406]
  Mask Diffusion Models (MDMs) naturally supports explicit local edits on previous outputs.<n>We propose Reflective Masking (RM), which elicits intrinsic reasoning capability in MDMs via lightweight post-training.
  arXiv  Detail & Related papers  (2026-06-15T13:34:38Z)
• Infinite Mask Diffusion for Few-Step Distillation [16.340935269530842]
  Masked Diffusion Models (MDMs) have emerged as a promising alternative to autoregressive models in language modeling.<n>We propose the Infinite Mask Diffusion Model (IMDM) to mitigate the theoretical bound while inheriting the benefits of MDMs.<n>IMDM surpasses existing few-step distillation methods at small step counts on LM1B and OpenWebText.
  arXiv  Detail & Related papers  (2026-05-11T13:07:03Z)
• On the Trainability of Masked Diffusion Language Models via Blockwise Locality [58.92209096047332]
  Masked diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large language models (AR-LLMs)<n>We study blockwise MDMs and compare them with AR-LLMs on three controlled tasks that stress different aspects of structured generation.<n>We find that standard random-masking MDMs fail to reliably learn linear regression, exhibit high variance training dynamics on graph path-finding, while outperforming AR-LLMs on Sudoku.
  arXiv  Detail & Related papers  (2026-04-27T17:44:26Z)
• Unifying Masked Diffusion Models with Various Generation Orders and Beyond [56.70289720766803]
  Masked diffusion models (MDMs) are a potential alternative to autoregressive models (ARMs) for language generation.<n>We propose order-expressive masked diffusion model (OeMDM) for a broad class of diffusion generative processes.<n>We introduce learnable-order masked diffusion model (LoMDM) which jointly learns the generation ordering and diffusion backbone.
  arXiv  Detail & Related papers  (2026-02-02T13:54:32Z)
• Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow [30.201913054064363]
  Masked Diffusion Language Models promise parallel token generation and arbitrary-order decoding.<n>We characterize MDLM behavior along two dimensions -- parallelism strength and generation order.<n>We evaluate eight mainstream MDLMs on 58 benchmarks spanning knowledge, reasoning, and programming.
  arXiv  Detail & Related papers  (2026-01-22T02:39:36Z)
• Masked Diffusion Models are Secretly Learned-Order Autoregressive Models [21.17429712617749]
  We show that Masked Diffusion Models can identify and optimize for a decoding order during training.<n>We prove that the MDM objective decomposes precisely into a weighted auto-regressive losses over these orders.
  arXiv  Detail & Related papers  (2025-11-24T14:17:56Z)
• Beyond Surface Reasoning: Unveiling the True Long Chain-of-Thought Capacity of Diffusion Large Language Models [54.81955614221652]
  parallel decoding, which enables simultaneous token updates, conflicts with the causal order often required for rigorous reasoning.<n> Behavioral analyses in both simple and complex reasoning tasks show thatDLLMs exhibit genuine parallelism only for directly decidable outputs.<n>We propose several practical mitigations, parallel-oriented prompting, diffusion early stopping, and parallel scaling, to reduce PSC-induced ineffectiveness and inefficiencies.
  arXiv  Detail & Related papers  (2025-10-10T16:58:14Z)
• Any-Order Flexible Length Masked Diffusion [53.89217188409148]
  Masked diffusion models (MDMs) have recently emerged as a promising alternative to autoregressive models over discrete domains.<n>We introduce Flexible Masked Diffusion Models (FlexMDMs), a discrete diffusion paradigm that simultaneously can model sequences of flexible length.<n>We show that FlexMDMs match MDMs in perplexity while modeling length statistics with much higher fidelity.
  arXiv  Detail & Related papers  (2025-08-31T23:34:53Z)
• Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs [11.261099213520158]
  Multimodal Sentiment Analysis (MSA) is a rapidly developing field that integrates multimodal information to recognize sentiments.<n>The central challenge in MSA is multimodal fusion, which is predominantly addressed by Multimodal Transformers (MulTs)<n>In this work, we propose and prove that MulTs are hierarchical modal-wise heterogeneous graphs (HMHGs) and we introduce the graph-structured representation pattern of MulTs.<n>Based on this pattern, we propose an Interlaced Mask (IM) mechanism to design the Graph-Structured and Interlaced-Masked Multimodal Transformer (GsiT)
  arXiv  Detail & Related papers  (2025-05-02T07:18:00Z)
• Efficient Reasoning with Hidden Thinking [48.96945580741641]
  Chain-of-Thought (CoT) reasoning has become a powerful framework for improving complex problem-solving capabilities.<n>We propose $textbfHeima$ (as hidden llama), an efficient reasoning framework that leverages reasoning CoTs at hidden latent space.<n>Heima model achieves higher generation efficiency while maintaining or even better zero-shot task accuracy.
  arXiv  Detail & Related papers  (2025-01-31T15:10:29Z)
• R-MTLLMF: Resilient Multi-Task Large Language Model Fusion at the Wireless Edge [78.26352952957909]
  Multi-task large language models (MTLLMs) are important for many applications at the wireless edge, where users demand specialized models to handle multiple tasks efficiently.<n>The concept of model fusion via task vectors has emerged as an efficient approach for combining fine-tuning parameters to produce an MTLLM.<n>In this paper, the problem of enabling edge users to collaboratively craft such MTLMs via tasks vectors is studied, under the assumption of worst-case adversarial attacks.
  arXiv  Detail & Related papers  (2024-11-27T10:57:06Z)
• Scaling up Masked Diffusion Models on Text [43.16800764711572]
  Masked diffusion models (MDMs) have shown promise in language modeling.<n>This paper establishes the first scaling law for MDMs.<n>We train a family of MDMs with up to 1.1 billion (B) parameters to evaluate their performance against larger sizes.
  arXiv  Detail & Related papers  (2024-10-24T08:01:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.