Differences in Text Generated by Diffusion and Autoregressive Language Models

• URL: http://arxiv.org/abs/2605.12522v1
• Date: Sat, 04 Apr 2026 17:30:35 GMT
• Title: Differences in Text Generated by Diffusion and Autoregressive Language Models
• Abstract summary: Diffusion language models (DLMs) are promising alternatives to autoregressive language models (ARMs)<n>We first find empirically that off-the-shelf DLMs exhibit lower $n$-gram entropy, higher semantic coherence, and higher semantic diversity.<n>We conduct experiments that decouple the effects of training objectives and decoding algorithms.
• Score: 21.816538128891256
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Diffusion language models (DLMs) are promising alternatives to autoregressive language models (ARMs), yet the intrinsic differences in their generated text remain underexplored. We first find empirically that off-the-shelf DLMs exhibit lower $n$-gram entropy, higher semantic coherence, and higher semantic diversity. To understand the cause, we conduct controlled experiments that decouple the effects of training objectives and decoding algorithms. Results suggest that the DLM training objective contributes to the increases in semantic coherence and semantic diversity, but has a minor influence on entropy. These differences are primarily driven by the bidirectional context; other components in the training objective, such as input masking, label masking, and the weighting function, have a much weaker influence. Further, our experiments demonstrate that the reduction in entropy stems from DLMs' decoding algorithms, particularly confidence-based remasking strategies. We provide a theoretical understanding for this entropy reduction phenomenon. Together, our work uncovers key mechanisms underlying the differences between DLMs and ARMs in text generation, and informs future design of training objectives and decoding algorithms in DLMs.

Related papers

• Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domain [94.64605746540472]
  Multimodal large language models (MLLMs) have extended the capability of large language models (LLMs) to process more contextual multimodal information.<n>Recent studies reveal that MLLMs remain highly vulnerable to adversarial inputs, especially those targeting visual components.<n>We propose a novel phase-aware adversarial attack framework that explicitly restricts adversarial perturbations to structure-relevant phase regions.
  arXiv  Detail & Related papers  (2026-09-01T06:34:32Z)
• Diffusion Language Models: An Experimental Analysis [21.773483524326362]
  Diffusion Language Models generate text through iterative denoising rather than next-token prediction.<n>We evaluate eight state-of-the-art DLMs across eight benchmarks spanning reasoning, coding, translation, knowledge, and structured problem solving.<n>We show that the behavior of DLMs is strongly influenced by generation-time design choices, leading to distinct trade-offs between performance and computational efficiency.
  arXiv  Detail & Related papers  (2026-06-17T18:10:23Z)
• Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design [59.05127237532803]
  Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models.<n>We study this sensitivity through a principled analysis of generalization error and identify three critical factors.<n>We propose SemDLM+, which adds a global transition and a semantic-frequency penalty during sampling.
  arXiv  Detail & Related papers  (2026-06-13T14:41:55Z)
• Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs [100.02824137397464]
  We investigate how Large Language Models adapt their internal representations when encountering inputs of increasing difficulty.<n>We reveal a consistent and quantifiable phenomenon: as task difficulty increases, the last hidden states of LLMs become substantially sparser.<n>This sparsity--difficulty relation is observable across diverse models and domains.
  arXiv  Detail & Related papers  (2026-03-03T18:48:15Z)
• PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMs [59.78917775399492]
  Multimodal instruction fine-tuning paradoxically degrades this text's reasoning capability.<n>We propose a training-free framework to mitigate this degradation.
  arXiv  Detail & Related papers  (2026-01-12T15:27:51Z)
• Reproducing and Dissecting Denoising Language Models for Speech Recognition [31.91567892562116]
  Denoising language models (DLMs) have been proposed as a powerful alternative to traditional language models (LMs) for automatic speech recognition (ASR)<n>This paper presents the first independent, large-scale empirical study of DLMs.
  arXiv  Detail & Related papers  (2025-12-15T17:33:22Z)
• Attention Sinks in Diffusion Language Models [15.450369268824835]
  Masked Diffusion Language Models (DLMs) have recently emerged as a promising alternative to traditional Autoregressive Models (ARMs)<n>We conduct an empirical analysis of DLM attention patterns, focusing on the attention sinking phenomenon, an effect previously observed in various transformer-based architectures.<n>Our findings reveal that DLMs also exhibit attention sinks, but with distinct characteristics. First, unlike in ARMs, the sink positions in DLMs tend to shift throughout the generation process, displaying a dynamic behaviour.
  arXiv  Detail & Related papers  (2025-10-17T15:23:58Z)
• PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning [54.73049408950049]
  We propose a Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning.<n>Our approach improves unified multimodal retrieval from both structural and learning perspectives.
  arXiv  Detail & Related papers  (2025-07-10T16:47:25Z)
• Odysseus Navigates the Sirens' Song: Dynamic Focus Decoding for Factual and Diverse Open-Ended Text Generation [18.835969818281125]
  Large Language Models (LLMs) are increasingly required to generate text that is both factually accurate and diverse across various open-ended applications.<n>We introduce Dynamic Focus Decoding (DFD), a novel plug-and-play approach that resolves this trade-off without requiring additional data, knowledge, or models.<n>DFD adaptively adjusts the decoding focus based on distributional differences across layers, leveraging the modular and hierarchical nature of factual knowledge within LLMs.
  arXiv  Detail & Related papers  (2025-03-11T05:27:28Z)
• Unified Generative and Discriminative Training for Multi-modal Large Language Models [88.84491005030316]
  Generative training has enabled Vision-Language Models (VLMs) to tackle various complex tasks. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval. This paper proposes a unified approach that integrates the strengths of both paradigms.
  arXiv  Detail & Related papers  (2024-11-01T01:51:31Z)
• Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary [65.268245109828]
  Non-human-like behaviour of contemporary pre-trained language models (PLMs) is a leading cause undermining their trustworthiness. A striking phenomenon is the generation of inconsistent predictions, which produces contradictory results. We propose a practical approach that alleviates the inconsistent behaviour issue by improving PLM awareness.
  arXiv  Detail & Related papers  (2023-10-24T06:15:15Z)
• A Cohesive Distillation Architecture for Neural Language Models [0.0]
  A recent trend in Natural Language Processing is the exponential growth in Language Model (LM) size. This study investigates methods for Knowledge Distillation (KD) to provide efficient alternatives to large-scale models.
  arXiv  Detail & Related papers  (2023-01-12T08:01:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.