Less-to-More Generalization: Unlocking More Controllability by In-Context Generation

• URL: http://arxiv.org/abs/2504.02160v1
• Date: Wed, 02 Apr 2025 22:20:21 GMT
• Title: Less-to-More Generalization: Unlocking More Controllability by In-Context Generation
• Authors: Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, Qian He,
• Abstract summary: We propose a highly-consistent data synthesis pipeline to tackle subject-driven generation challenges.<n>This pipeline harnesses the intrinsic in-context generation capabilities of diffusion transformers and generates high-consistency multi-subject paired data.<n>We also introduce UNO, which consists of progressive cross-modal alignment and universal rotary position embedding.
• Score: 4.832184187988317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although subject-driven generation has been extensively explored in image generation due to its wide applications, it still has challenges in data scalability and subject expansibility. For the first challenge, moving from curating single-subject datasets to multiple-subject ones and scaling them is particularly difficult. For the second, most recent methods center on single-subject generation, making it hard to apply when dealing with multi-subject scenarios. In this study, we propose a highly-consistent data synthesis pipeline to tackle this challenge. This pipeline harnesses the intrinsic in-context generation capabilities of diffusion transformers and generates high-consistency multi-subject paired data. Additionally, we introduce UNO, which consists of progressive cross-modal alignment and universal rotary position embedding. It is a multi-image conditioned subject-to-image model iteratively trained from a text-to-image model. Extensive experiments show that our method can achieve high consistency while ensuring controllability in both single-subject and multi-subject driven generation.

Related papers

• The Power of Context: How Multimodality Improves Image Super-Resolution [42.21009967392721]
  Single-image super-resolution (SISR) remains challenging due to the inherent difficulty of recovering fine-grained details from low-resolution inputs.<n>We propose a novel approach that leverages the rich contextual information available in multiple modalities to learn a powerful generative prior for SISR.<n>Our model surpasses state-of-the-art generative SISR methods, achieving superior visual quality and fidelity.
  arXiv  Detail & Related papers  (2025-03-18T17:59:54Z)
• UniCombine: Unified Multi-Conditional Combination with Diffusion Transformer [24.159791066104358]
  We introduce a DiT-based multi-conditional controllable generative framework capable of handling any combination of conditions.<n>Specifically, we introduce a novel MMDiT Attention mechanism and incorporate a trainable LoRA module.<n>We also propose a new pipeline to construct SubjectSpatial200K, the first dataset designed for multi-conditional generative tasks.
  arXiv  Detail & Related papers  (2025-03-12T11:22:47Z)
• OminiControl: Minimal and Universal Control for Diffusion Transformer [68.3243031301164]
  We present OminiControl, a novel approach that rethinks how image conditions are integrated into Diffusion Transformer (DiT) architectures.<n>OminiControl addresses these limitations through three key innovations.
  arXiv  Detail & Related papers  (2024-11-22T17:55:15Z)
• A Simple Approach to Unifying Diffusion-based Conditional Generation [63.389616350290595]
  We introduce a simple, unified framework to handle diverse conditional generation tasks.<n>Our approach enables versatile capabilities via different inference-time sampling schemes.<n>Our model supports additional capabilities like non-spatially aligned and coarse conditioning.
  arXiv  Detail & Related papers  (2024-10-15T09:41:43Z)
• OneActor: Consistent Character Generation via Cluster-Conditioned Guidance [29.426558840522734]
  We propose a novel one-shot tuning paradigm, termed OneActor. It efficiently performs consistent subject generation solely driven by prompts. Our method is capable of multi-subject generation and compatible with popular diffusion extensions.
  arXiv  Detail & Related papers  (2024-04-16T03:45:45Z)
• Multi-View Unsupervised Image Generation with Cross Attention Guidance [23.07929124170851]
  This paper introduces a novel pipeline for unsupervised training of a pose-conditioned diffusion model on single-category datasets. We identify object poses by clustering the dataset through comparing visibility and locations of specific object parts. Our model, MIRAGE, surpasses prior work in novel view synthesis on real images.
  arXiv  Detail & Related papers  (2023-12-07T14:55:13Z)
• StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data [129.92449761766025]
  We propose a novel data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. This approach harnesses the power of generative models, marrying the abilities of ChatGPT and text-to-image generative models. Our research includes comprehensive experiments conducted on various datasets.
  arXiv  Detail & Related papers  (2023-08-20T12:43:52Z)
• Mutual Information-driven Triple Interaction Network for Efficient Image Dehazing [54.168567276280505]
  We propose a novel Mutual Information-driven Triple interaction Network (MITNet) for image dehazing. The first stage, named amplitude-guided haze removal, aims to recover the amplitude spectrum of the hazy images for haze removal. The second stage, named phase-guided structure refined, devotes to learning the transformation and refinement of the phase spectrum.
  arXiv  Detail & Related papers  (2023-08-14T08:23:58Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)
• Latent Processes Identification From Multi-View Time Series [17.33428123777779]
  We propose a novel framework that employs the contrastive learning technique to invert the data generative process for enhanced identifiability. MuLTI integrates a permutation mechanism that merges corresponding overlapped variables by the establishment of an optimal transport formula.
  arXiv  Detail & Related papers  (2023-05-14T14:21:58Z)
• DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation [56.514462874501675]
  We propose a dynamic sparse attention based Transformer model to achieve fine-level matching with favorable efficiency. The heart of our approach is a novel dynamic-attention unit, dedicated to covering the variation on the optimal number of tokens one position should focus on. Experiments on three applications, pose-guided person image generation, edge-based face synthesis, and undistorted image style transfer, demonstrate that DynaST achieves superior performance in local details.
  arXiv  Detail & Related papers  (2022-07-13T11:12:03Z)
• Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Models [54.1843419649895]
  A major challenge in using generative models to accomplish this task is the lack of paired data containing all modalities and corresponding outputs. We propose a solution based on a denoising diffusion probabilistic synthesis models to generate images under multi-model priors.
  arXiv  Detail & Related papers  (2022-06-10T12:23:05Z)
• Text Generation with Deep Variational GAN [16.3190206770276]
  We propose a GAN-based generic framework to address the problem of mode-collapse in a principled approach. We show that our model can generate realistic text with high diversity.
  arXiv  Detail & Related papers  (2021-04-27T21:42:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.