UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer

• URL: http://arxiv.org/abs/2606.16255v1
• Date: Mon, 15 Jun 2026 05:57:40 GMT
• Title: UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer
• Abstract summary: Unified Multimodal Models (UMMs) have emerged as a critical direction for general-purpose multimodal intelligence.<n>Existing UMMs face prominent challenges: (1) the inherent learning conflicts between visual understanding and generation tasks, leading to suboptimal modeling in both tasks; (2) different understanding and generation visual spaces impeding scalability; (3) over-reliance on task-specific data that neglects the duality of text-image understanding and generation.
• Score: 29.975180930024067
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unified Multimodal Models (UMMs) have emerged as a critical direction for general-purpose multimodal intelligence, integrating understanding and generation into a single framework. However, existing UMMs face prominent challenges: (1) the inherent learning conflicts between visual understanding and generation tasks, leading to suboptimal modeling in both tasks; (2) different understanding and generation visual spaces impeding scalability; (3) over-reliance on task-specific data that neglects the duality of text-image understanding and generation. To address these challenges, we propose UniDDT, which leverages a Noisy ViT encoder along with an LLM to unify semantic encoding for visual generation and understanding tasks, while employing a separate diffusion decoder to decouple diffusion decoding from text decoding. With this Noisy ViT encoder, UniDDT is able to leverage the latent space as a unified visual representation, enabling seamless compatibility between understanding and generation tasks. Thus, the scalability within the generation tasks and the semantic expressiveness within understanding tasks can be balanced. Also, we construct dual data structures from the same image-text pairs, fostering interdependence between the generation and understanding data to exploit their inherent duality. Extensive experiments demonstrate that UniDDT achieves effective unification of multimodal understanding and generation with enhanced semantic consistency and scalability. For visual generation tasks, our UniDDT achieves 0.87 GenEval score and 86.9 DPG overall score. For multimodal understanding tasks, our UniDDT achieves 1699.5 score on MME benchmark and 76.5 overall score on SEEDbench.

Related papers

• Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System [90.64481646805855]
  unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model.<n>We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting.<n>We show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.
  arXiv  Detail & Related papers  (2026-09-01T17:59:59Z)
• Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models [98.8608163448532]
  unified multimodal models (UMMs) have made remarkable progress in integrating visual understanding and generation.<n>We propose a token-level intrinsic text-image alignment reward mechanism, GvU, enabling the UMM to act simultaneously as teacher and student.<n>We show that our method substantially boosts UMMs' generation, which in turn strengthens their fine-grained visual understanding.
  arXiv  Detail & Related papers  (2026-03-06T08:56:14Z)
• TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models [96.41974190202642]
  Unified multimodal models (UMMs) aim to jointly perform multimodal understanding and generation within a single framework.<n>We present TUNA, a native UMM that builds a unified continuous visual representation by cascading a VAE encoder with a representation encoder.
  arXiv  Detail & Related papers  (2025-12-01T18:59:51Z)
• UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception [54.53657134205492]
  UniAlignment is a unified multimodal generation framework within a single diffusion transformer.<n>It incorporates both intrinsic-modal semantic alignment and cross-modal semantic alignment, thereby enhancing the model's cross-modal consistency and instruction-following robustness.<n>We present SemGen-Bench, a new benchmark specifically designed to evaluate multimodal semantic consistency under complex textual instructions.
  arXiv  Detail & Related papers  (2025-09-28T09:11:30Z)
• UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets [51.284864284520744]
  Unified vision large language models (VLLMs) have recently achieved impressive advancements in both multimodal understanding and generation.<n>We introduce a novel dataset construction framework, UnifiedVisual, and present UnifiedVisual-240K.<n>UnifiedVisual-240K seamlessly integrates diverse visual and textual inputs and outputs, enabling comprehensive cross-modal reasoning.
  arXiv  Detail & Related papers  (2025-09-18T08:39:44Z)
• UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation [39.921363034430875]
  Unified image understanding and generation has emerged as a promising paradigm in multimodal artificial intelligence.<n>We study the modality alignment behaviors of task-specific expert models for understanding and generation.<n>We introduce UniFork, a novel Y-shaped architecture that shares the shallow layers for cross-task representation learning, while employing task-specific branches in deeper layers to avoid task interference.
  arXiv  Detail & Related papers  (2025-06-20T17:52:31Z)
• Harmonizing Visual Representations for Unified Multimodal Understanding and Generation [53.01486796503091]
  We present emphHarmon, a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder.<n>Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T20:50:38Z)
• DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies [25.77487827338777]
  A vision tokenizer trained for reconstruction excels at capturing low-level perceptual details.<n>A vision encoder trained via contrastive learning aligns well with language but struggles to decode back into the pixel space for generation tasks.<n>We propose DualToken, a method that unifies representations for both understanding and generation within a single tokenizer.
  arXiv  Detail & Related papers  (2025-03-18T14:56:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.