MagicFace: Training-free Universal-Style Human Image Customized Synthesis

• URL: http://arxiv.org/abs/2408.07433v5
• Date: Mon, 18 Nov 2024 03:14:37 GMT
• Title: MagicFace: Training-free Universal-Style Human Image Customized Synthesis
• Authors: Yibin Wang, Weizhong Zhang, Cheng Jin,
• Abstract summary: MagicFace is a training-free method for multi-concept universal-style human image personalized synthesis. Our core idea is to simulate how humans create images given specific concepts, first establish a semantic layout. In the first stage, RSA enables the latent image to query features from all reference concepts simultaneously.
• Score: 13.944050414488911
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current human image customization methods leverage Stable Diffusion (SD) for its rich semantic prior. However, since SD is not specifically designed for human-oriented generation, these methods often require extensive fine-tuning on large-scale datasets, which renders them susceptible to overfitting and hinders their ability to personalize individuals with previously unseen styles. Moreover, these methods extensively focus on single-concept human image synthesis and lack the flexibility to customize individuals using multiple given concepts, thereby impeding their broader practical application. This paper proposes MagicFace, a novel training-free method for multi-concept universal-style human image personalized synthesis. Our core idea is to simulate how humans create images given specific concepts, i.e., first establish a semantic layout considering factors such as concepts' shape and posture, then optimize details by comparing with concepts at the pixel level. To implement this process, we introduce a coarse-to-fine generation pipeline, involving two sequential stages: semantic layout construction and concept feature injection. This is achieved by our Reference-aware Self-Attention (RSA) and Region-grouped Blend Attention (RBA) mechanisms. In the first stage, RSA enables the latent image to query features from all reference concepts simultaneously, extracting the overall semantic understanding to facilitate the initial semantic layout establishment. In the second stage, we employ an attention-based semantic segmentation method to pinpoint the latent generated regions of all concepts at each step. Following this, RBA divides the pixels of the latent image into semantic groups, with each group querying fine-grained features from the corresponding reference concept. Extensive experiments demonstrate the superiority of our MagicFace.

Related papers

• Per-Query Visual Concept Learning [32.045160884721646]
  We show that many existing methods can be substantially augmented by adding a personalization step.<n>Specifically, we leverage PDM features - previously designed to capture identity - and show how they can be used to improve semantic similarity.
  arXiv  Detail & Related papers  (2025-08-12T16:07:27Z)
• OmniPrism: Learning Disentangled Visual Concept for Image Generation [57.21097864811521]
  Creative visual concept generation often draws inspiration from specific concepts in a reference image to produce relevant outcomes. We propose OmniPrism, a visual concept disentangling approach for creative image generation. Our method learns disentangled concept representations guided by natural language and trains a diffusion model to incorporate these concepts.
  arXiv  Detail & Related papers  (2024-12-16T18:59:52Z)
• AttenCraft: Attention-guided Disentanglement of Multiple Concepts for Text-to-Image Customization [4.544788024283586]
  AttenCraft is an attention-guided method for multiple concept disentanglement. We introduce Uniform sampling and Reweighted sampling schemes to alleviate the non-synchronicity of feature acquisition from different concepts. Our method outperforms baseline models in terms of image-alignment, and behaves comparably on text-alignment.
  arXiv  Detail & Related papers  (2024-05-28T08:50:14Z)
• FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition [49.2208591663092]
  FreeCustom is a tuning-free method to generate customized images of multi-concept composition based on reference concepts. We introduce a new multi-reference self-attention (MRSA) mechanism and a weighted mask strategy. Our method outperforms or performs on par with other training-based methods in terms of multi-concept composition and single-concept customization.
  arXiv  Detail & Related papers  (2024-05-22T17:53:38Z)
• MC$^2$: Multi-concept Guidance for Customized Multi-concept Generation [59.00909718832648]
  We propose MC$2$, a novel approach for multi-concept customization. By adaptively refining attention weights between visual and textual tokens, our method ensures that image regions accurately correspond to their associated concepts. Experiments demonstrate that MC$2$ outperforms training-based methods in terms of prompt-reference alignment.
  arXiv  Detail & Related papers  (2024-04-08T07:59:04Z)
• Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis [65.7968515029306]
  We propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for Pose-Guided Person Image Synthesis (PGPIS) A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt.
  arXiv  Detail & Related papers  (2024-02-28T06:07:07Z)
• Visual Concept-driven Image Generation with Text-to-Image Diffusion Model [65.96212844602866]
  Text-to-image (TTI) models have demonstrated impressive results in generating high-resolution images of complex scenes. Recent approaches have extended these methods with personalization techniques that allow them to integrate user-illustrated concepts. However, the ability to generate images with multiple interacting concepts, such as human subjects, as well as concepts that may be entangled in one, or across multiple, image illustrations remains illusive. We propose a concept-driven TTI personalization framework that addresses these core challenges.
  arXiv  Detail & Related papers  (2024-02-18T07:28:37Z)
• Break-A-Scene: Extracting Multiple Concepts from a Single Image [80.47666266017207]
  We introduce the task of textual scene decomposition. We propose augmenting the input image with masks that indicate the presence of target concepts. We then present a novel two-phase customization process.
  arXiv  Detail & Related papers  (2023-05-25T17:59:04Z)
• Designing an Encoder for Fast Personalization of Text-to-Image Models [57.62449900121022]
  We propose an encoder-based domain-tuning approach for text-to-image personalization. We employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively ingest additional concepts.
  arXiv  Detail & Related papers  (2023-02-23T18:46:41Z)
• HumanDiffusion: a Coarse-to-Fine Alignment Diffusion Framework for Controllable Text-Driven Person Image Generation [73.3790833537313]
  Controllable person image generation promotes a wide range of applications such as digital human interaction and virtual try-on. We propose HumanDiffusion, a coarse-to-fine alignment diffusion framework, for text-driven person image generation.
  arXiv  Detail & Related papers  (2022-11-11T14:30:34Z)
• ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentation [17.019848796027485]
  Self-supervised visual pre-training models have shown great promise in representing pixel-level semantic relationships. In this work, we investigate the pixel-level semantic aggregation in self-trained models as image encodes and design concepts. We propose the Adaptive Concept Generator (ACG) which adaptively maps these prototypes to informative concepts for each image.
  arXiv  Detail & Related papers  (2022-10-12T06:16:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.