Layer Collapse in Diffusion Language Models

• URL: http://arxiv.org/abs/2605.06366v2
• Date: Mon, 11 May 2026 08:56:53 GMT
• Title: Layer Collapse in Diffusion Language Models
• Abstract summary: Diffusion language models (DLMs) have emerged as competitive alternatives to autoregressive (AR) language models.<n>We show that layer collapse in DLMs is not driven by undertraining but by overtraining.<n>Our findings have strong practical implications.
• Score: 54.880703002010144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion language models (DLMs) have recently emerged as competitive alternatives to autoregressive (AR) language models, yet differences in their activation dynamics remain poorly understood. We characterize these dynamics in LLaDA-8B and identify a striking layer-collapse property: a few early layers exhibit highly similar, collapsed activation patterns dominated by a single large super-outlier persisting over a long token range. Despite its apparent redundancy, this outlier is critical: pruning it causes outputs to degrade into repetitive random token loops. Paradoxically, layers in LLaDA contain more redundant representations overall, with redundancy most pronounced in earlier layers -- the reverse of AR models, where deeper layers grow redundant due to undertraining. Our analysis indicates that layer collapse in DLMs is not driven by undertraining but by overtraining: a dominant outlier becomes an indispensable information carrier while remaining representations collapse into redundant structure. These findings have strong practical implications, verified through controlled pre-training experiments. DLMs are surprisingly robust to compression: LLaDA under 3-bit GPTQ quantization drops only -1.8% on GSM8K, whereas Llama-3.1-8B drops -64.7%. Optimal sparsity allocation also reverses between families: at 50% average sparsity, allocating more to early layers in LLaDA yields +8.4% over the reverse strategy, while the same allocation costs Llama -8.4%. Our findings reveal that the DLM training objective fundamentally reshapes layer dynamics relative to AR models, with direct consequences for compression and deployment. Code: github.com/Conzel/super-outlier-dlm.

Related papers

• Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training [17.723489440201277]
  Reinforcement learning (RL) has become a central component of post-training large language models.<n>We study how RL adaptation is distributed across transformer layers.<n>We observe a remarkably stable pattern: RL gains are highly concentrated in a small subset, and in many cases even a single, transformer layers.
  arXiv  Detail & Related papers  (2026-07-01T17:59:54Z)
• Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding [92.1521161575198]
  We introduce Confident Decoding, a training-free decoding strategy that dynamically selects the most reliable near-final layer.<n> Experiments across dense and Mixture-of-Experts LLMs demonstrate consistent gains on challenging reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-06-20T07:03:26Z)
• Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs [32.78500862748985]
  Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes.<n>In this work, we reveal a surprising phenomenon: ZO fine-tuning is sharply dominated by a single decoding layer.<n>Experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks show that dominant-layer ZO fine-tuning improves average performance over full-model MeZO and LoRA-based ZO fine-tuning.
  arXiv  Detail & Related papers  (2026-06-03T23:42:09Z)
• Skip to the Good Part: Representation Structure & Inference-Time Layer Skipping in Diffusion vs. Autoregressive LLMs [39.20818218608441]
  Autoregressive (AR) language models form representations incrementally through left-to-right prediction, whereas diffusion language models (dLLMs) are trained via full-sequence denoising.<n>We find that diffusion objectives result in different, more hierarchical abstractions with substantial early-layer redundancy and reduced recency bias.<n>We introduce a static, task-agnostic inference-time layer-skipping method requiring no architectural changes or KV-cache sharing.<n>Native dLLMs achieve up to 18.75% FLOPs reduction while preserving over 90% performance on reasoning and code generation benchmarks, whereas AR models degrade sharply under comparable skipping
  arXiv  Detail & Related papers  (2026-03-08T05:31:52Z)
• On GRPO Collapse in Search-R1: The Lazy Likelihood-Displacement Death Spiral [59.14787085809595]
  We identify Lazy Likelihood Displacement (LLD) as the core mechanism driving this failure.<n>LLD emerges early and triggers a self-reinforcing LLD Death Spiral, where declining likelihood leads to low-confidence responses.<n>We propose a lightweight likelihood-preserving regularization LLDS for GRPO that activates only when a trajectory's likelihood decreases.
  arXiv  Detail & Related papers  (2025-12-03T19:41:15Z)
• TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compression [55.37723860832064]
  We propose a dynamic ratio-based training pipeline that does not rely on sophisticated data annotations.<n>We validate our approach across models on DeepSeek-R1-Distill-7B and DeepSeek-R1-Distill-14B and on a diverse set of benchmarks with varying difficulty levels.
  arXiv  Detail & Related papers  (2025-06-03T09:23:41Z)
• DLP: Dynamic Layerwise Pruning in Large Language Models [20.810186827378434]
  Pruning has been widely adopted to reduce the parameter scale and improve the inference efficiency of Large Language Models (LLMs)<n>We propose a novel method called Dynamic Layerwise Pruning (DLP)<n>This approach adaptively determines the relative importance of each layer by integrating model weights with input activation information, assigning pruning rates accordingly.
  arXiv  Detail & Related papers  (2025-05-27T07:35:00Z)
• LESA: Learnable LLM Layer Scaling-Up [57.0510934286449]
  Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive.<n>Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones.<n>We propose textbfLESA, a novel learnable method for depth scaling-up.
  arXiv  Detail & Related papers  (2025-02-19T14:58:48Z)
• Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding [74.31981011985681]
  Large language models (LLMs) have shown impressive capabilities, but still struggle with complex reasoning tasks requiring multiple steps. We introduce LaTent Reasoning Optimization (LaTRO), a principled framework that formulates reasoning as sampling from a latent distribution. We validate LaTRO through experiments on GSM8K and ARC-Challenge datasets using multiple model architectures.
  arXiv  Detail & Related papers  (2024-11-06T22:02:30Z)
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect [38.148626520751385]
  We show that many layers of Large Language Models (LLMs) exhibit high similarity, and some layers play a negligible role in network functionality. We propose a straightforward pruning approach: layer removal, in which we directly delete the redundant layers. Experiments demonstrate that our method, which we call ShortGPT, significantly outperforms previous state-of-the-art (SOTA) methods in model pruning.
  arXiv  Detail & Related papers  (2024-03-06T17:04:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.