MagicFace: Training-free Universal-Style Human Image Customized Synthesis

• URL: http://arxiv.org/abs/2408.07433v3
• Date: Mon, 19 Aug 2024 14:43:24 GMT
• Title: MagicFace: Training-free Universal-Style Human Image Customized Synthesis
• Authors: Yibin Wang, Weizhong Zhang, Cheng Jin,
• Abstract summary: MagicFace is a training-free method for universal-style human image personalized synthesis. It integrates reference concept features into their latent generated region at the pixel level. Experiments demonstrate the superiority of MagicFace in both human-centric subject-to-image synthesis and multi-concept human image customization.
• Score: 13.944050414488911
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current state-of-the-art methods for human image customized synthesis typically require tedious training on large-scale datasets. In such cases, they are prone to overfitting and struggle to personalize individuals of unseen styles. Moreover, these methods extensively focus on single-concept human image synthesis and lack the flexibility needed for customizing individuals with multiple given concepts, thereby impeding their broader practical application. To this end, we propose MagicFace, a novel training-free method for universal-style human image personalized synthesis, enabling multi-concept customization by accurately integrating reference concept features into their latent generated region at the pixel level. Specifically, MagicFace introduces a coarse-to-fine generation pipeline, involving two sequential stages: semantic layout construction and concept feature injection. This is achieved by our Reference-aware Self-Attention (RSA) and Region-grouped Blend Attention (RBA) mechanisms. In the first stage, RSA enables the latent image to query features from all reference concepts simultaneously, extracting the overall semantic understanding to facilitate the initial semantic layout establishment. In the second stage, we employ an attention-based semantic segmentation method to pinpoint the latent generated regions of all concepts at each step. Following this, RBA divides the pixels of the latent image into semantic groups, with each group querying fine-grained features from the corresponding reference concept, which ensures precise attribute alignment and feature injection. Throughout the generation process, a weighted mask strategy is employed to ensure the model focuses more on the reference concepts. Extensive experiments demonstrate the superiority of MagicFace in both human-centric subject-to-image synthesis and multi-concept human image customization.

Related papers

• OmniPrism: Learning Disentangled Visual Concept for Image Generation [57.21097864811521]
  Creative visual concept generation often draws inspiration from specific concepts in a reference image to produce relevant outcomes. We propose OmniPrism, a visual concept disentangling approach for creative image generation. Our method learns disentangled concept representations guided by natural language and trains a diffusion model to incorporate these concepts.
  arXiv  Detail & Related papers  (2024-12-16T18:59:52Z)
• AttenCraft: Attention-guided Disentanglement of Multiple Concepts for Text-to-Image Customization [4.544788024283586]
  AttenCraft is an attention-guided method for multiple concept disentanglement. We introduce Uniform sampling and Reweighted sampling schemes to alleviate the non-synchronicity of feature acquisition from different concepts. Our method outperforms baseline models in terms of image-alignment, and behaves comparably on text-alignment.
  arXiv  Detail & Related papers  (2024-05-28T08:50:14Z)
• FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition [49.2208591663092]
  FreeCustom is a tuning-free method to generate customized images of multi-concept composition based on reference concepts. We introduce a new multi-reference self-attention (MRSA) mechanism and a weighted mask strategy. Our method outperforms or performs on par with other training-based methods in terms of multi-concept composition and single-concept customization.
  arXiv  Detail & Related papers  (2024-05-22T17:53:38Z)
• MC$^2$: Multi-concept Guidance for Customized Multi-concept Generation [59.00909718832648]
  We propose MC$2$, a novel approach for multi-concept customization. By adaptively refining attention weights between visual and textual tokens, our method ensures that image regions accurately correspond to their associated concepts. Experiments demonstrate that MC$2$ outperforms training-based methods in terms of prompt-reference alignment.
  arXiv  Detail & Related papers  (2024-04-08T07:59:04Z)
• Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis [65.7968515029306]
  We propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for Pose-Guided Person Image Synthesis (PGPIS) A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt.
  arXiv  Detail & Related papers  (2024-02-28T06:07:07Z)
• Visual Concept-driven Image Generation with Text-to-Image Diffusion Model [65.96212844602866]
  Text-to-image (TTI) models have demonstrated impressive results in generating high-resolution images of complex scenes. Recent approaches have extended these methods with personalization techniques that allow them to integrate user-illustrated concepts. However, the ability to generate images with multiple interacting concepts, such as human subjects, as well as concepts that may be entangled in one, or across multiple, image illustrations remains illusive. We propose a concept-driven TTI personalization framework that addresses these core challenges.
  arXiv  Detail & Related papers  (2024-02-18T07:28:37Z)
• Break-A-Scene: Extracting Multiple Concepts from a Single Image [80.47666266017207]
  We introduce the task of textual scene decomposition. We propose augmenting the input image with masks that indicate the presence of target concepts. We then present a novel two-phase customization process.
  arXiv  Detail & Related papers  (2023-05-25T17:59:04Z)
• Designing an Encoder for Fast Personalization of Text-to-Image Models [57.62449900121022]
  We propose an encoder-based domain-tuning approach for text-to-image personalization. We employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively ingest additional concepts.
  arXiv  Detail & Related papers  (2023-02-23T18:46:41Z)
• HumanDiffusion: a Coarse-to-Fine Alignment Diffusion Framework for Controllable Text-Driven Person Image Generation [73.3790833537313]
  Controllable person image generation promotes a wide range of applications such as digital human interaction and virtual try-on. We propose HumanDiffusion, a coarse-to-fine alignment diffusion framework, for text-driven person image generation.
  arXiv  Detail & Related papers  (2022-11-11T14:30:34Z)
• ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentation [17.019848796027485]
  Self-supervised visual pre-training models have shown great promise in representing pixel-level semantic relationships. In this work, we investigate the pixel-level semantic aggregation in self-trained models as image encodes and design concepts. We propose the Adaptive Concept Generator (ACG) which adaptively maps these prototypes to informative concepts for each image.
  arXiv  Detail & Related papers  (2022-10-12T06:16:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.