UniCombine: Unified Multi-Conditional Combination with Diffusion Transformer

• URL: http://arxiv.org/abs/2503.09277v1
• Date: Wed, 12 Mar 2025 11:22:47 GMT
• Title: UniCombine: Unified Multi-Conditional Combination with Diffusion Transformer
• Authors: Haoxuan Wang, Jinlong Peng, Qingdong He, Hao Yang, Ying Jin, Jiafu Wu, Xiaobin Hu, Yanjie Pan, Zhenye Gan, Mingmin Chi, Bo Peng, Yabiao Wang,
• Abstract summary: We introduce a DiT-based multi-conditional controllable generative framework capable of handling any combination of conditions.<n>Specifically, we introduce a novel MMDiT Attention mechanism and incorporate a trainable LoRA module.<n>We also propose a new pipeline to construct SubjectSpatial200K, the first dataset designed for multi-conditional generative tasks.
• Score: 24.159791066104358
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid development of diffusion models in image generation, the demand for more powerful and flexible controllable frameworks is increasing. Although existing methods can guide generation beyond text prompts, the challenge of effectively combining multiple conditional inputs while maintaining consistency with all of them remains unsolved. To address this, we introduce UniCombine, a DiT-based multi-conditional controllable generative framework capable of handling any combination of conditions, including but not limited to text prompts, spatial maps, and subject images. Specifically, we introduce a novel Conditional MMDiT Attention mechanism and incorporate a trainable LoRA module to build both the training-free and training-based versions. Additionally, we propose a new pipeline to construct SubjectSpatial200K, the first dataset designed for multi-conditional generative tasks covering both the subject-driven and spatially-aligned conditions. Extensive experimental results on multi-conditional generation demonstrate the outstanding universality and powerful capability of our approach with state-of-the-art performance.

Related papers

• Less-to-More Generalization: Unlocking More Controllability by In-Context Generation [4.832184187988317]
  We propose a highly-consistent data synthesis pipeline to tackle subject-driven generation challenges. This pipeline harnesses the intrinsic in-context generation capabilities of diffusion transformers and generates high-consistency multi-subject paired data. We also introduce UNO, which consists of progressive cross-modal alignment and universal rotary position embedding.
  arXiv  Detail & Related papers  (2025-04-02T22:20:21Z)
• Unified Multimodal Discrete Diffusion [78.48930545306654]
  Multimodal generative models that can understand and generate across multiple modalities are dominated by autoregressive (AR) approaches. We explore discrete diffusion models as a unified generative formulation in the joint text and image domain. We present the first Unified Multimodal Discrete Diffusion (UniDisc) model which is capable of jointly understanding and generating text and images.
  arXiv  Detail & Related papers  (2025-03-26T17:59:51Z)
• MMGen: Unified Multi-modal Image Generation and Understanding in One Go [60.97155790727879]
  We introduce MMGen, a unified framework that integrates multiple generative tasks into a single diffusion model. Our approach develops a novel diffusion transformer that flexibly supports multi-modal output, along with a simple modality-decoupling strategy.
  arXiv  Detail & Related papers  (2025-03-26T15:37:17Z)
• FullDiT: Multi-Task Video Generative Foundation Model with Full Attention [37.776430879317765]
  FullDiT is a unified foundation model for video generation that seamlessly integrates multiple conditions via unified full-attention mechanisms. Experiments demonstrate that FullDiT achieves state-of-the-art results, highlighting the efficacy of full-attention in complex multi-task video generation.
  arXiv  Detail & Related papers  (2025-03-25T17:59:06Z)
• EasyControl: Adding Efficient and Flexible Control for Diffusion Transformer [15.879712910520801]
  We propose EasyControl, a novel framework designed to unify condition-guided diffusion transformers with high efficiency and flexibility.<n>Our framework is built on three key innovations. First, we introduce a lightweight Condition Injection LoRA Module.<n>Second, we propose a Position-Aware Training Paradigm. This approach standardizes input conditions to fixed resolutions, allowing the generation of images with arbitrary aspect ratios and flexible resolutions.<n>Third, we develop a Causal Attention Mechanism combined with the KV Cache technique, adapted for conditional generation tasks.
  arXiv  Detail & Related papers  (2025-03-10T08:07:17Z)
• MINT: Multi-modal Chain of Thought in Unified Generative Models for Enhanced Image Generation [38.517814177255765]
  We introduce MINT, an innovative unified generative model, empowered with native multimodal chain of thought (MCoT) for enhanced image generation.<n>We propose an innovative MCoT training paradigm, a step-by-step approach to multimodal thinking, reasoning, and reflection specifically designed to enhance image generation.<n> MINT has been validated to exhibit superior performance across multiple benchmarks for text-to-image (T2I) and image-to-text (I2T) tasks.
  arXiv  Detail & Related papers  (2025-03-03T08:36:16Z)
• OminiControl: Minimal and Universal Control for Diffusion Transformer [68.3243031301164]
  We present OminiControl, a novel approach that rethinks how image conditions are integrated into Diffusion Transformer (DiT) architectures.<n>OminiControl addresses these limitations through three key innovations.
  arXiv  Detail & Related papers  (2024-11-22T17:55:15Z)
• A Simple Approach to Unifying Diffusion-based Conditional Generation [63.389616350290595]
  We introduce a simple, unified framework to handle diverse conditional generation tasks. Our approach enables versatile capabilities via different inference-time sampling schemes. Our model supports additional capabilities like non-spatially aligned and coarse conditioning.
  arXiv  Detail & Related papers  (2024-10-15T09:41:43Z)
• EMMA: Your Text-to-Image Diffusion Model Can Secretly Accept Multi-Modal Prompts [48.214475133206385]
  EMMA is a novel image generation model accepting multi-modal prompts built upon the state-of-the-art text-to-image (T2I) diffusion model, ELLA. By freezing all parameters in the original T2I diffusion model and only adjusting some additional layers, we reveal an interesting finding that the pre-trained T2I diffusion model can secretly accept multi-modal prompts.
  arXiv  Detail & Related papers  (2024-06-13T14:26:43Z)
• Combinatorial Multivariant Multi-Armed Bandits with Applications to Episodic Reinforcement Learning and Beyond [58.39457881271146]
  We introduce a novel framework of multi-armed bandits (CMAB) with multivariant and probabilistically triggering arms (CMAB-MT) Compared with existing CMAB works, CMAB-MT not only enhances the modeling power but also allows improved results by leveraging distinct statistical properties for multivariant random variables. Our framework can include many important problems as applications, such as episodic reinforcement learning (RL) and probabilistic maximum coverage for goods distribution.
  arXiv  Detail & Related papers  (2024-06-03T14:48:53Z)
• MA-ViT: Modality-Agnostic Vision Transformers for Face Anti-Spoofing [3.3031006227198003]
  We present Modality-Agnostic Vision Transformer (MA-ViT), which aims to improve the performance of arbitrary modal attacks with the help of multi-modal data. Specifically, MA-ViT adopts the early fusion to aggregate all the available training modalities data and enables flexible testing of any given modal samples. Experiments demonstrate that the single model trained on MA-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin.
  arXiv  Detail & Related papers  (2023-04-15T13:03:44Z)
• mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video [89.19867891570945]
  mPLUG-2 is a new unified paradigm with modularized design for multi-modal pretraining. It shares common universal modules for modality collaboration and disentangling different modality modules to deal with modality entanglement. It is flexible to select different modules for different understanding and generation tasks across all modalities including text, image, and video.
  arXiv  Detail & Related papers  (2023-02-01T12:40:03Z)
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers [108.92194081987967]
  We make the first attempt to explore a universal multi-agent reinforcement learning pipeline, designing one single architecture to fit tasks. Unlike previous RNN-based models, we utilize a transformer-based model to generate a flexible policy. The proposed model, named as Universal Policy Decoupling Transformer (UPDeT), further relaxes the action restriction and makes the multi-agent task's decision process more explainable.
  arXiv  Detail & Related papers  (2021-01-20T07:24:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.