Unveiling the Secret of AdaLN-Zero in Diffusion Transformer

• URL: http://arxiv.org/abs/2608.09438v1
• Date: Mon, 10 Aug 2026 11:12:17 GMT
• Title: Unveiling the Secret of AdaLN-Zero in Diffusion Transformer
• Abstract summary: Diffusion transformer (DiT) is a rapidly emerging architecture for image generation.<n>We investigate a critical conditioning mechanism within DiT, adaLN-Zero, which achieves superior performance compared to adaLN.<n>We introduce an improved conditioning mechanism called SE-adaLN-Zero.
• Score: 54.52619563900124
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Diffusion transformer (DiT), a rapidly emerging architecture for image generation, has gained much attention. However, despite ongoing efforts to improve its performance, the understanding of DiT remains superficial. In this work, we delve into and investigate a critical conditioning mechanism within DiT, adaLN-Zero, which achieves superior performance compared to adaLN. Our work studies three potential elements driving this performance, including an SE-like structure, zero-initialization, and a "gradual" update order, among which zero-initialization is proved to be the most influential. Building on this understanding, we propose an analysis-guided initialization strategy, termed adaLN-Gaussian, which serves both as an empirical validation of our analysis and as a practical initialization method that consistently improves optimization efficiency. On the other hand, inspired by the SE-like structure, we introduce an improved conditioning mechanism called SE-adaLN-Zero. Extensive experiments following DiT on four datasets, especially on ImageNet1K demonstrate the effectiveness and generalization of adaLN-Gaussian and SE-adaLN-Zero. Beyond class-to-image generation, we also evaluate the generalization of the two improved methods on text-to-image generation.

Related papers

• Maximizing Incremental Information Entropy for Contrastive Learning [42.45787033288968]
  We propose IE-CL (Incremental-Entropy Contrastive Learning), a framework that explicitly optimize the entropy gain between augmented views.<n>Our theoretical framework reframes the challenge by identifying an information encoder and proposes a joint optimization of two components.<n>Experiments on CIFAR-10/100, STL-10, and ImageNet demonstrate that IE-CL consistently improves performance under small-batch settings.
  arXiv  Detail & Related papers  (2026-03-13T02:51:37Z)
• Implicit Neural Representation-Based Continuous Single Image Super Resolution: An Empirical Study [50.15623093332659]
  Implicit neural representation (INR) has become the standard approach for arbitrary-scale image super-resolution (ASSR)<n>We compare existing techniques across diverse settings and present aggregated performance results on multiple image quality metrics.<n>We examine a new loss function that penalizes intensity variations while preserving edges, textures, and finer details during training.
  arXiv  Detail & Related papers  (2026-01-25T07:09:20Z)
• NOVAK: Unified adaptive optimizer for deep neural networks [0.0]
  NOVAK is a gradient-based optimization algorithm that integrates adaptive moment estimation, rectified learning-rate scheduling, decoupled weight regularization, multiple variants of Nesterov momentum, and lookahead synchronization into a unified, performance-oriented framework.
  arXiv  Detail & Related papers  (2026-01-11T13:03:57Z)
• How Does Fourier Analysis Network Work? A Mechanism Analysis and a New Dual-Activation Layer Proposal [0.0]
  We show that only the sine activation contributes positively to performance, whereas the cosine activation tends to be detrimental.<n>FAN primarily alleviates the dying-ReLU problem, in which a neuron consistently receives negative inputs, produces zero gradients, and stops learning.
  arXiv  Detail & Related papers  (2025-12-16T19:36:56Z)
• Decoupled Global-Local Alignment for Improving Compositional Understanding [19.022748171561357]
  Contrastive Language-Image Pre-training (CLIP) has achieved success on multiple downstream tasks by aligning image and text modalities.<n>However, the nature of global contrastive learning limits CLIP's ability to comprehend compositional concepts.<n>We introduce a Decoupled Global-Local Alignment (DeGLA) framework that improves compositional understanding while substantially mitigating losses in general capabilities.
  arXiv  Detail & Related papers  (2025-04-23T15:20:53Z)
• Efficient Diffusion as Low Light Enhancer [63.789138528062225]
  Reflectance-Aware Trajectory Refinement (RATR) is a simple yet effective module to refine the teacher trajectory using the reflectance component of images. textbfReflectance-aware textbfDiffusion with textbfDistilled textbfTrajectory (textbfReDDiT) is an efficient and flexible distillation framework tailored for Low-Light Image Enhancement (LLIE)
  arXiv  Detail & Related papers  (2024-10-16T08:07:18Z)
• Non-asymptotic Convergence of Training Transformers for Next-token Prediction [48.9399496805422]
  Transformers have achieved extraordinary success in modern machine learning due to their excellent ability to handle sequential data. This paper provides a fine-grained non-asymptotic analysis of the training dynamics of a one-layer transformer. We show that the trained transformer presents non-token prediction ability with dataset shift.
  arXiv  Detail & Related papers  (2024-09-25T20:22:06Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• DiffSCI: Zero-Shot Snapshot Compressive Imaging via Iterative Spectral Diffusion Model [18.25548360119976]
  This paper endeavors to advance the precision of snapshot compressive imaging (SCI) reconstruction for multispectral image (MSI) We propose a novel structured zero-shot diffusion model, dubbed DiffSCI. We present extensive testing to show that DiffSCI exhibits discernible performance enhancements over prevailing self-supervised and zero-shot approaches.
  arXiv  Detail & Related papers  (2023-11-19T20:27:14Z)
• Toward Fast, Flexible, and Robust Low-Light Image Enhancement [87.27326390675155]
  We develop a new Self-Calibrated Illumination (SCI) learning framework for fast, flexible, and robust brightening images in real-world low-light scenarios. Considering the computational burden of the cascaded pattern, we construct the self-calibrated module which realizes the convergence between results of each stage. We make comprehensive explorations to SCI's inherent properties including operation-insensitive adaptability and model-irrelevant generality.
  arXiv  Detail & Related papers  (2022-04-21T14:40:32Z)
• Iterative Network for Image Super-Resolution [69.07361550998318]
  Single image super-resolution (SISR) has been greatly revitalized by the recent development of convolutional neural networks (CNN) This paper provides a new insight on conventional SISR algorithm, and proposes a substantially different approach relying on the iterative optimization. A novel iterative super-resolution network (ISRN) is proposed on top of the iterative optimization.
  arXiv  Detail & Related papers  (2020-05-20T11:11:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.