MC$^2$: Multi-concept Guidance for Customized Multi-concept Generation

• URL: http://arxiv.org/abs/2404.05268v3
• Date: Sat, 30 Nov 2024 11:55:19 GMT
• Title: MC$^2$: Multi-concept Guidance for Customized Multi-concept Generation
• Authors: Jiaxiu Jiang, Yabo Zhang, Kailai Feng, Xiaohe Wu, Wenbo Li, Renjing Pei, Fan Li, Wangmeng Zuo,
• Abstract summary: We propose MC$2$, a novel approach for multi-concept customization. By adaptively refining attention weights between visual and textual tokens, our method ensures that image regions accurately correspond to their associated concepts. Experiments demonstrate that MC$2$ outperforms training-based methods in terms of prompt-reference alignment.
• Score: 59.00909718832648
• License:
• Abstract: Customized text-to-image generation, which synthesizes images based on user-specified concepts, has made significant progress in handling individual concepts. However, when extended to multiple concepts, existing methods often struggle with properly integrating different models and avoiding the unintended blending of characteristics from distinct concepts. In this paper, we propose MC$^2$, a novel approach for multi-concept customization that enhances flexibility and fidelity through inference-time optimization. MC$^2$ enables the integration of multiple single-concept models with heterogeneous architectures. By adaptively refining attention weights between visual and textual tokens, our method ensures that image regions accurately correspond to their associated concepts while minimizing interference between concepts. Extensive experiments demonstrate that MC$^2$ outperforms training-based methods in terms of prompt-reference alignment. Furthermore, MC$^2$ can be seamlessly applied to text-to-image generation, providing robust compositional capabilities. To facilitate the evaluation of multi-concept customization, we also introduce a new benchmark, MC++. The code will be publicly available at https://github.com/JIANGJiaXiu/MC-2.

Related papers

• ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty [52.15933752463479]
  ConceptMix is a scalable, controllable, and customizable benchmark. It automatically evaluates compositional generation ability of Text-to-Image (T2I) models. It reveals that the performance of several models, especially open models, drops dramatically with increased k.
  arXiv  Detail & Related papers  (2024-08-26T15:08:12Z)
• Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis [14.21719970175159]
  Concept Conductor is designed to ensure visual fidelity and correct layout in multi-concept customization. We present a concept injection technique that employs shape-aware masks to specify the generation area for each concept. Our method supports the combination of any number of concepts and maintains high fidelity even when dealing with visually similar concepts.
  arXiv  Detail & Related papers  (2024-08-07T08:43:58Z)
• FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition [49.2208591663092]
  FreeCustom is a tuning-free method to generate customized images of multi-concept composition based on reference concepts. We introduce a new multi-reference self-attention (MRSA) mechanism and a weighted mask strategy. Our method outperforms or performs on par with other training-based methods in terms of multi-concept composition and single-concept customization.
  arXiv  Detail & Related papers  (2024-05-22T17:53:38Z)
• MultiBooth: Towards Generating All Your Concepts in an Image from Text [29.02126551676985]
  This paper introduces MultiBooth, a novel and efficient technique for multi-concept customization in image generation from text. In the single-concept learning phase, we employ a multi-modal image encoder and an efficient concept encoding technique to learn a concise and discriminative representation for each concept. In the multi-concept integration phase, we use bounding boxes to define the generation area for each concept within the cross-attention map.
  arXiv  Detail & Related papers  (2024-04-22T14:47:54Z)
• Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition [47.07564907486087]
  Recent text-to-image diffusion models are able to learn and synthesize images containing novel, personalized concepts. This paper tackles two interconnected issues within this realm of personalizing text-to-image diffusion models.
  arXiv  Detail & Related papers  (2024-02-23T18:55:09Z)
• Visual Concept-driven Image Generation with Text-to-Image Diffusion Model [65.96212844602866]
  Text-to-image (TTI) models have demonstrated impressive results in generating high-resolution images of complex scenes. Recent approaches have extended these methods with personalization techniques that allow them to integrate user-illustrated concepts. However, the ability to generate images with multiple interacting concepts, such as human subjects, as well as concepts that may be entangled in one, or across multiple, image illustrations remains illusive. We propose a concept-driven TTI personalization framework that addresses these core challenges.
  arXiv  Detail & Related papers  (2024-02-18T07:28:37Z)
• Break-A-Scene: Extracting Multiple Concepts from a Single Image [80.47666266017207]
  We introduce the task of textual scene decomposition. We propose augmenting the input image with masks that indicate the presence of target concepts. We then present a novel two-phase customization process.
  arXiv  Detail & Related papers  (2023-05-25T17:59:04Z)
• Multi-Concept Customization of Text-to-Image Diffusion [51.8642043743222]
  We propose Custom Diffusion, an efficient method for augmenting existing text-to-image models. We find that only optimizing a few parameters in the text-to-image conditioning mechanism is sufficiently powerful to represent new concepts. Our model generates variations of multiple new concepts and seamlessly composes them with existing concepts in novel settings.
  arXiv  Detail & Related papers  (2022-12-08T18:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.