DiverseDiT: Towards Diverse Representation Learning in Diffusion Transformers

• URL: http://arxiv.org/abs/2603.04239v1
• Date: Wed, 04 Mar 2026 16:21:42 GMT
• Title: DiverseDiT: Towards Diverse Representation Learning in Diffusion Transformers
• Abstract summary: We show that representation diversity across blocks is a crucial factor for effective learning in Diffusion Transformers (DiTs)<n>We propose DiverseDiT, a novel framework that explicitly promotes representation diversity.<n>Our work provides valuable insights into the representation learning dynamics of DiTs and offers a practical approach for enhancing their performance.
• Score: 18.873998808314067
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent breakthroughs in Diffusion Transformers (DiTs) have revolutionized the field of visual synthesis due to their superior scalability. To facilitate DiTs' capability of capturing meaningful internal representations, recent works such as REPA incorporate external pretrained encoders for representation alignment. However, the underlying mechanisms governing representation learning within DiTs are not well understood. To this end, we first systematically investigate the representation dynamics of DiTs. Through analyzing the evolution and influence of internal representations under various settings, we reveal that representation diversity across blocks is a crucial factor for effective learning. Based on this key insight, we propose DiverseDiT, a novel framework that explicitly promotes representation diversity. DiverseDiT incorporates long residual connections to diversify input representations across blocks and a representation diversity loss to encourage blocks to learn distinct features. Extensive experiments on ImageNet 256x256 and 512x512 demonstrate that our DiverseDiT yields consistent performance gains and convergence acceleration when applied to different backbones with various sizes, even when tested on the challenging one-step generation setting. Furthermore, we show that DiverseDiT is complementary to existing representation learning techniques, leading to further performance gains. Our work provides valuable insights into the representation learning dynamics of DiTs and offers a practical approach for enhancing their performance.

Related papers

• DiverseDiT++: Quantifying, Analyzing, and Promoting Representation Diversity in Diffusion Transformers [48.96981796971673]
  We show that representation diversity across blocks is a critical factor for effective representation learning in DiTs.<n>We propose DiverseDiT++, a novel framework that explicitly promotes diverse representation learning.<n>Our method incorporates long residual connections to diversify input representations across blocks and a representation diversity loss to encourage blocks to learn distinct features.
  arXiv  Detail & Related papers  (2026-08-04T03:51:15Z)
• ViQ: Text-Aligned Visual Quantized Representations at Any Resolution [91.46185855575789]
  We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations.<n>Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization.<n>Experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders.
  arXiv  Detail & Related papers  (2026-06-25T17:29:27Z)
• Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation [81.40978077888693]
  Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance.<n>Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens.<n>We integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations.
  arXiv  Detail & Related papers  (2026-03-05T04:45:49Z)
• Learning Task-Agnostic Representations through Multi-Teacher Distillation [59.488314181423284]
  We introduce a task-agnostic framework based on a majority vote" objective function.<n>We demonstrate that this function is bounded by the mutual information between student and teachers' embeddings.<n>Our method effectively leverages teacher diversity, resulting in representations enabling better performance for a wide range of downstream tasks.
  arXiv  Detail & Related papers  (2025-10-21T14:36:33Z)
• PartFormer: Awakening Latent Diverse Representation from Vision Transformer for Object Re-Identification [73.64560354556498]
  Vision Transformer (ViT) tends to overfit on most distinct regions of training data, limiting its generalizability and attention to holistic object features. We present PartFormer, an innovative adaptation of ViT designed to overcome the limitations in object Re-ID tasks. Our framework significantly outperforms state-of-the-art by 2.4% mAP scores on the most challenging MSMT17 dataset.
  arXiv  Detail & Related papers  (2024-08-29T16:31:05Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• Learning Expressive Prompting With Residuals for Vision Transformers [11.342913284654706]
  We present Expressive Prompts with Residuals (EXPRES) which modifies the prompt learning paradigm specifically for effective adaptation of vision transformers (ViT) We apply EXPRES for image classification, few shot learning, and semantic segmentation, and show our method is capable of achieving state of the art prompt tuning on 3/3 categories of the VTAB benchmark.
  arXiv  Detail & Related papers  (2023-03-27T20:47:01Z)
• Semantics-Consistent Feature Search for Self-Supervised Visual Representation Learning [15.242064747740116]
  It is unavoidable to construct undesirable views containing different semantic concepts during the augmentation procedure. It would damage the semantic consistency of representation to pull these augmentations closer in the feature space indiscriminately. In this study, we introduce feature-level augmentation and propose a novel semantics-consistent feature search (SCFS) method to mitigate this negative effect.
  arXiv  Detail & Related papers  (2022-12-13T11:13:59Z)
• Demystify Transformers & Convolutions in Modern Image Deep Networks [80.16624587948368]
  This paper aims to identify the real gains of popular convolution and attention operators through a detailed study.<n>We find that the key difference among these feature transformation modules, such as attention or convolution, lies in their spatial feature aggregation approach.<n>Various STMs are integrated into this unified framework for comprehensive comparative analysis.
  arXiv  Detail & Related papers  (2022-11-10T18:59:43Z)
• Improving the Modality Representation with Multi-View Contrastive Learning for Multimodal Sentiment Analysis [15.623293264871181]
  This study investigates the improvement approaches of modality representation with contrastive learning. We devise a three-stages framework with multi-view contrastive learning to refine representations for the specific objectives. We conduct experiments on three open datasets, and results show the advance of our model.
  arXiv  Detail & Related papers  (2022-10-28T01:25:16Z)
• Investigating the Properties of Neural Network Representations in Reinforcement Learning [35.02223992335008]
  This paper empirically investigates the properties of representations that support transfer in reinforcement learning. We consider Deep Q-learning agents with different auxiliary losses in a pixel-based navigation environment. We develop a method to better understand why some representations work better for transfer, through a systematic approach.
  arXiv  Detail & Related papers  (2022-03-30T00:14:26Z)
• Deep Partial Multi-View Learning [94.39367390062831]
  We propose a novel framework termed Cross Partial Multi-View Networks (CPM-Nets) We fifirst provide a formal defifinition of completeness and versatility for multi-view representation. We then theoretically prove the versatility of the learned latent representations.
  arXiv  Detail & Related papers  (2020-11-12T02:29:29Z)
• Improving Transformation Invariance in Contrastive Representation Learning [31.223892428863238]
  We introduce a training objective for contrastive learning that uses a novel regularizer to control how the representation changes under transformation. Second, we propose a change to how test time representations are generated by introducing a feature averaging approach that combines encodings from multiple transformations of the original input. Third, we introduce the novel Spirograph dataset to explore our ideas in the context of a differentiable generative process with multiple downstream tasks.
  arXiv  Detail & Related papers  (2020-10-19T13:49:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.