Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers

• URL: http://arxiv.org/abs/2605.06169v1
• Date: Thu, 07 May 2026 12:53:45 GMT
• Title: Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers
• Abstract summary: We show that networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation.<n>We propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement.
• Score: 0.2436681150766912
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.

Related papers

• Uncertainty DMD: Restoring Diversity in Few-Step Autoregressive Video Distillation [52.34982929108336]
  Few-step distillation improves the efficiency of autoregressive (AR) video generation, but often causes diversity collapse.<n>We analyze this degradation in Distribution Matching Distillation (DMD)-distilled AR video generators and find that, in the autoregressive setting, it takes the form of a structured uncertainty collapse.
  arXiv  Detail & Related papers  (2026-09-10T08:59:47Z)
• Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation [7.440412390819483]
  Diversify, Anchor, and Filter (DAF) is a framework that augments entropy-based adaptation with a marginal diversity loss that resists collapse.<n>We evaluate on five datasets spanning natural scenes, autonomous driving, underwater imagery, and remote sensing with their corrupted variants.
  arXiv  Detail & Related papers  (2026-08-30T17:39:14Z)
• SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models [53.86918766240095]
  Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
  arXiv  Detail & Related papers  (2026-08-23T10:41:07Z)
• Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure [8.63727823701713]
  Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants.<n>We show how causal attention creates high-similarity representations and why training dynamics fail to repair them.
  arXiv  Detail & Related papers  (2026-08-10T10:45:16Z)
• Deep Slice Interpolation for Reducing Through-Plane Anisotropy and Noise in Head CT [0.0]
  We present a deep learning system that synthesizes intermediate CT slices from pairs of neighboring axial slices.<n>The system improves three-dimensional visualization while simultaneously producing inherently denoised outputs.
  arXiv  Detail & Related papers  (2026-06-08T10:45:51Z)
• A lift for input-convex neural network training [3.142113135607563]
  In Inputdimensional neural networks (ICNNs) are used for logconcave density estimation, convexpotential normalizing flows, optimal transport, and transportmap inversion for posteriors.<n>The standard projected gradient descent (PGD) onto the non-negative cone applies a hard, nonsmooth projection.<n>The differentiable alternative, softplus reparametrization, attenuates the gradient exponentially in the weight magnitude, stalling training with dead interlayer weights loss.<n>We show that the lift reaches a lower test loss than both PGD and direct softplus, and turns a plateau-bounded training into a
  arXiv  Detail & Related papers  (2026-05-22T22:59:11Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• scHelix: Asymmetric Dual-Stream Integration via Explicit Gene-Level Disentanglement [84.88065404629079]
  scHelix is a dataset-adaptive framework that fundamentally changes how features are processed.<n> scHelix utilizes a dual-stream sparse diffusion encoder equipped with stop-gradient graph caching.<n> scHelix outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-18T15:55:13Z)
• Escaping Mode Collapse in LLM Generation via Geometric Regulation [11.183390901786659]
  We take a dynamical-systems view and reinterpret mode collapse as reduced state-space accessibility.<n>We proposeReinforced Mode Regulation (RMR), a lightweight, online state-space intervention that regulates dominant self-reinforcing directions.<n>RMR substantially reduces mode collapse and enables stable, high-quality generation at extremely low entropy rates.
  arXiv  Detail & Related papers  (2026-05-01T06:12:05Z)
• Convergence of Byzantine-Resilient Gradient Tracking via Probabilistic Edge Dropout [1.3902537392439644]
  We study distributed optimization over networks with Byzantine agents that may send arbitrary adversarial messages.<n>We propose emphGradient Tracking with Probabilistic Edge Dropout and Leaky Integration (GT-PD-L)<n>GT-PD-L outperforms coordinate-wise mean by up to 4.3 percentage points under stealth attacks.
  arXiv  Detail & Related papers  (2026-04-01T03:55:42Z)
• Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inference [58.189320101488725]
  DLLMs promise fast non-autoregressive inference but suffer a severe quality-speed trade-off in parallel decoding.<n>We address this by integrating continuous representations into the discrete decoding process, as they preserve rich inter-position dependency.<n>We propose ReMix, a framework that introduces a novel Continuous Mixing State as an intermediate between the initial masked state and the final decoded token state.
  arXiv  Detail & Related papers  (2026-02-26T11:08:11Z)
• From Circuits to Dynamics: Understanding and Stabilizing Failure in 3D Diffusion Transformers [25.11520870904882]
  3D diffusion transformers exhibit a catastrophic mode of failure.<n>We call this phenomenon Meltdown.<n>We introduce PowerRemap, a test-time control that stabilizes sparse point-cloud conditioning.
  arXiv  Detail & Related papers  (2026-02-11T18:42:05Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• Improving Vector-Quantized Image Modeling with Latent Consistency-Matching Diffusion [55.185588994883226]
  We introduce VQ-LCMD, a continuous-space latent diffusion framework within the embedding space that stabilizes training.<n>VQ-LCMD uses a novel training objective combining the joint embedding-diffusion variational lower bound with a consistency-matching (CM) loss.<n>Experiments show that the proposed VQ-LCMD yields superior results on FFHQ, LSUN Churches, and LSUN Bedrooms compared to discrete-state latent diffusion models.
  arXiv  Detail & Related papers  (2024-10-18T09:12:33Z)
• Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization [68.49738668084693]
  Self-supervised pre-training recently demonstrates success on large-scale multimodal data. Cross-modality alignment (CMA) is only a weak and noisy supervision. CMA might cause conflicts and biases among modalities.
  arXiv  Detail & Related papers  (2022-11-03T18:12:32Z)
• Dynamic melting and condensation of topological dislocation modes [0.0]
  Bulk dislocation lattice defects are instrumental in identifying translationally active topological insulators (TATIs) We show that when a TATI via a real time ramp enters into a trivial or translationally inert insulating phase, the signatures of the preramp defect modes survive for a long time. Proposed dynamic responses at the dislocation core can be experimentally observed in quantum crystals, optical lattices and metamaterials with time a tunable band gap.
  arXiv  Detail & Related papers  (2022-10-27T17:59:27Z)
• Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment [52.02794488304448]
  We propose a new distribution alignment method based on a log-likelihood ratio statistic and normalizing flows. We experimentally verify that minimizing the resulting objective results in domain alignment that preserves the local structure of input domains.
  arXiv  Detail & Related papers  (2020-03-26T22:10:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.