UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception

• URL: http://arxiv.org/abs/2606.30332v1
• Date: Mon, 29 Jun 2026 14:13:27 GMT
• Title: UniGP: Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception
• Abstract summary: UniGP is a framework built upon MMDiT that unifies controllable generation and dense prediction.<n>By learning controllable generation and prediction under different conditions, our model effectively captures the joint distribution of image-geometry pairs.
• Score: 21.921528706051205
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in diffusion models have shown impressive performance in controllable image generation and dense prediction tasks. However, existing approaches typically treat diffusion-based controllable generation and dense prediction as separate tasks, overlooking the potential benefits of jointly modeling the heterogeneous distributions. In this work, we introduce UniGP, a framework built upon MMDiT, which unifies controllable generation and dense prediction through simple joint training, without the need for complex task-specific designs or losses, while preserving the backbone's versatile priors. By learning controllable generation and prediction under different conditions, our model effectively captures the joint distribution of image-geometry pairs. UniGP is capable of versatile controllable generation, dense prediction, and joint generation. Specifically, the proposed UniGP consists of DUGP and a unified dataset training strategy. The former, following the principle of Occam's razor, uses only a copied image branch of MMDiT to model dense distributions beyond RGB, while the latter integrates heterogeneous datasets into a unified training framework to jointly model generation and perception tasks. Extensive experiments demonstrate that our unified model surpasses prior unified approaches and performs on par with specialized methods. Furthermore, we demonstrate that multi-task joint training provides complementary benefits: generative priors enrich perceptual details, while perceptual learning improves structural alignment in generation.

Related papers

• UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation [51.41441081823758]
  Unified models capable of interleaved generation have emerged as a promising paradigm.<n>We propose a unified reinforcement learning framework tailored for interleaved generation.<n>Our experiments demonstrate that this unified training recipe significantly enhances image generation quality through reasoning.
  arXiv  Detail & Related papers  (2026-03-24T17:59:17Z)
• CMAD: Cooperative Multi-Agent Diffusion via Stochastic Optimal Control [40.52516377402657]
  Continuous-time generative models have achieved remarkable success in image restoration and synthesis.<n>Controlling the composition of multiple pre-trained models remains an open challenge.
  arXiv  Detail & Related papers  (2026-02-11T15:12:43Z)
• MMGRid: Navigating Temporal-aware and Cross-domain Generative Recommendation via Model Merging [22.681048070167765]
  Generative Recommendation (GR) has emerged as a new paradigm in recommender systems (RSs)<n>We focus on a fundamental yet underexplored challenge in real-world: how to merge generative recommenders specialized to different real-world contexts.<n>We propose a unified framework MMGRid, a structured contextual grid of GR checkpoints that organizes models trained under diverse contexts.
  arXiv  Detail & Related papers  (2026-01-22T13:09:16Z)
• MMaDA: Multimodal Large Diffusion Language Models [61.13527224215318]
  We introduce MMaDA, a novel class of multimodal diffusion foundation models.<n>It is designed to achieve superior performance across diverse domains such as textual reasoning, multimodal understanding, and text-to-image generation.
  arXiv  Detail & Related papers  (2025-05-21T17:59:05Z)
• Single Domain Generalization with Model-aware Parametric Batch-wise Mixup [22.709796153794507]
  Single Domain Generalization remains a formidable challenge in the field of machine learning.<n>We propose a novel data augmentation approach, named as Model-aware Parametric Batch-wise Mixup.<n>By exploiting inter-feature correlations, the parameterized mixup generator introduces additional versatility in combining features across a batch of instances.
  arXiv  Detail & Related papers  (2025-02-22T03:45:18Z)
• A Simple Approach to Unifying Diffusion-based Conditional Generation [63.389616350290595]
  We introduce a simple, unified framework to handle diverse conditional generation tasks.<n>Our approach enables versatile capabilities via different inference-time sampling schemes.<n>Our model supports additional capabilities like non-spatially aligned and coarse conditioning.
  arXiv  Detail & Related papers  (2024-10-15T09:41:43Z)
• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction [88.65168366064061]
  We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference. Our framework leads to a family of three novel objectives that are all simulation-free, and thus scalable. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.
  arXiv  Detail & Related papers  (2024-10-10T17:18:30Z)
• InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion [53.90516061351706]
  We present InterHandGen, a novel framework that learns the generative prior of two-hand interaction. For sampling, we combine anti-penetration and synthesis-free guidance to enable plausible generation. Our method significantly outperforms baseline generative models in terms of plausibility and diversity.
  arXiv  Detail & Related papers  (2024-03-26T06:35:55Z)
• Ensemble Modeling for Multimodal Visual Action Recognition [50.38638300332429]
  We propose an ensemble modeling approach for multimodal action recognition. We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset.
  arXiv  Detail & Related papers  (2023-08-10T08:43:20Z)
• Fast Ensemble Learning Using Adversarially-Generated Restricted Boltzmann Machines [0.0]
  Restricted Boltzmann Machine (RBM) has received recent attention and relies on an energy-based structure to model data probability distributions. This work proposes to artificially generate RBMs using Adversarial Learning, where pre-trained weight matrices serve as the GAN inputs. Experimental results demonstrate the suitability of the proposed approach under image reconstruction and image classification tasks.
  arXiv  Detail & Related papers  (2021-01-04T16:00:47Z)
• Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent [28.006781039853575]
  A key element behind the progress of machine learning in recent years has been the ability to train machine learning models in largescale distributed-memory environments. In this paper, we introduce general convergence methods used in practice to train large-scale machine learning models. Our framework, called elastic elastic bounds, enables us to derive convergence bounds for a variety of distributed SGD methods.
  arXiv  Detail & Related papers  (2020-01-16T16:10:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.