UniPortrait: A Unified Framework for Identity-Preserving Single- and Multi-Human Image Personalization

• URL: http://arxiv.org/abs/2408.05939v2
• Date: Fri, 6 Sep 2024 14:44:12 GMT
• Title: UniPortrait: A Unified Framework for Identity-Preserving Single- and Multi-Human Image Personalization
• Authors: Junjie He, Yifeng Geng, Liefeng Bo,
• Abstract summary: UniPortrait is an innovative human image personalization framework that unifies single- and multi-ID customization. UniPortrait consists of only two plug-and-play modules: an ID embedding module and an ID routing module.
• Score: 10.760799194716922
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents UniPortrait, an innovative human image personalization framework that unifies single- and multi-ID customization with high face fidelity, extensive facial editability, free-form input description, and diverse layout generation. UniPortrait consists of only two plug-and-play modules: an ID embedding module and an ID routing module. The ID embedding module extracts versatile editable facial features with a decoupling strategy for each ID and embeds them into the context space of diffusion models. The ID routing module then combines and distributes these embeddings adaptively to their respective regions within the synthesized image, achieving the customization of single and multiple IDs. With a carefully designed two-stage training scheme, UniPortrait achieves superior performance in both single- and multi-ID customization. Quantitative and qualitative experiments demonstrate the advantages of our method over existing approaches as well as its good scalability, e.g., the universal compatibility with existing generative control tools. The project page is at https://aigcdesigngroup.github.io/UniPortrait-Page/ .

Related papers

• Optimizing ID Consistency in Multimodal Large Models: Facial Restoration via Alignment, Entanglement, and Disentanglement [54.199726425201895]
  Multimodal editing large models have demonstrated powerful editing capabilities across diverse tasks.<n>Current facial ID preservation methods struggle to achieve consistent restoration of both facial identity and edited element IP.<n>We propose EditedID, an Alignment-Disentanglement-Entanglement framework for robust identity-specific facial restoration.
  arXiv  Detail & Related papers  (2026-02-21T08:24:42Z)
• Unified Personalized Understanding, Generating and Editing [54.5563878110386]
  We present textbf OmniPersona, an end-to-end personalization framework for unified LMMs.<n>It integrates personalized understanding, generation, and image editing within a single architecture.<n>Experiments demonstrate that OmniPersona delivers competitive and robust performance across diverse personalization tasks.
  arXiv  Detail & Related papers  (2026-01-11T15:46:34Z)
• Towards Generalized Multi-Image Editing for Unified Multimodal Models [56.620038824933566]
  Unified Multimodal Models (UMMs) integrate multimodal understanding and generation.<n>UMMs are limited to maintaining visual consistency and disambiguating visual cues when referencing details across multiple input images.<n>We propose a scalable multi-image editing framework for UMMs that explicitly distinguishes image identities and generalizes to variable input counts.
  arXiv  Detail & Related papers  (2026-01-09T06:42:49Z)
• UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation [54.38636515750502]
  We propose UniLayDiff: a Unified Diffusion Transformer for content-aware layout generation tasks.<n>We employ Multi-Modal Diffusion Transformer framework to capture the complex interplay between the background image, layout elements, and diverse constraints.<n>Experiments demonstrate that UniLayDiff achieves state-of-the-art performance across from unconditional to various conditional generation tasks.
  arXiv  Detail & Related papers  (2025-12-09T18:38:44Z)
• Training for Identity, Inference for Controllability: A Unified Approach to Tuning-Free Face Personalization [16.851646868288135]
  We introduce UniID, a unified tuning-free framework that synergistically integrates both paradigms.<n>Our key insight is that when merging these approaches, they should mutually reinforce only identity-relevant information.<n>This principled design enables UniID to achieve high-fidelity face personalization with flexible text controllability.
  arXiv  Detail & Related papers  (2025-12-03T16:57:50Z)
• A Training-Free Approach for Multi-ID Customization via Attention Adjustment and Spatial Control [7.810140287905315]
  Multi-ID customization is much more difficult and poses two major challenges.<n>It often encounters the copy-paste issue during inference, leading to lower quality.<n>We present an ID-decoupled cross-attention mechanism, injecting distinct ID embeddings into the corresponding image regions.
  arXiv  Detail & Related papers  (2025-11-25T15:28:10Z)
• UMO: Scaling Multi-Identity Consistency for Image Customization via Matching Reward [15.094319754425468]
  We present UMO, a framework designed to maintain high-fidelity identity preservation and alleviate identity confusion with scalability.<n>With "multi-to-multi matching" paradigm, UMO reformulates multi-identity generation as a global assignment optimization problem.<n>We develop a scalable customization dataset with multi-reference images, consisting of both synthesised and real parts.
  arXiv  Detail & Related papers  (2025-09-08T15:54:55Z)
• Concat-ID: Towards Universal Identity-Preserving Video Synthesis [23.40342294656802]
  We present Concat-ID, a unified framework for identity-preserving video synthesis. Concat-ID employs Autoencoders to extract image features, which are latent with video sequence latents. A novel cross-video pairing strategy and a multi-stage training regimen are introduced to balance consistency and facial editability.
  arXiv  Detail & Related papers  (2025-03-18T11:17:32Z)
• DynamicID: Zero-Shot Multi-ID Image Personalization with Flexible Facial Editability [12.692129257068085]
  DynamicID is a tuning-free framework supported by a dual-stage training paradigm. We have developed a curated VariFace-10k facial dataset, comprising 10k unique individuals, each represented by 35 distinct facial images.
  arXiv  Detail & Related papers  (2025-03-09T08:16:19Z)
• IC-Portrait: In-Context Matching for View-Consistent Personalized Portrait [51.18967854258571]
  IC-Portrait is a novel framework designed to accurately encode individual identities for personalized portrait generation. Our key insight is that pre-trained diffusion models are fast learners for in-context dense correspondence matching. We show that IC-Portrait consistently outperforms existing state-of-the-art methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2025-01-28T18:59:03Z)
• Omni-ID: Holistic Identity Representation Designed for Generative Tasks [75.29174595706533]
  Omni-ID encodes holistic information about an individual's appearance across diverse expressions. It consolidates information from a varied number of unstructured input images into a structured representation. It demonstrates substantial improvements over conventional representations across various generative tasks.
  arXiv  Detail & Related papers  (2024-12-12T19:21:20Z)
• CustAny: Customizing Anything from A Single Example [73.90939022698399]
  We present a novel pipeline to construct a large dataset of general objects, featuring 315k text-image samples across 10k categories. With the help of MC-IDC, we introduce Customizing Anything (CustAny), a zero-shot framework that maintains ID fidelity and supports flexible text editing for general objects. Our contributions include a large-scale dataset, the CustAny framework and novel ID processing to advance this field.
  arXiv  Detail & Related papers  (2024-06-17T15:26:22Z)
• InstantFamily: Masked Attention for Zero-shot Multi-ID Image Generation [0.0]
  "InstantFamily" is an approach that employs a novel cross-attention mechanism and a multimodal embedding stack to achieve zero-shot multi-ID image generation. Our method effectively preserves ID as it utilizes global and local features from a pre-trained face recognition model integrated with text conditions.
  arXiv  Detail & Related papers  (2024-04-30T10:16:21Z)
• ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with Reward Feedback Learning [57.91881829308395]
  Identity-preserving text-to-image generation (ID-T2I) has received significant attention due to its wide range of application scenarios like AI portrait and advertising. We present textbfID-Aligner, a general feedback learning framework to enhance ID-T2I performance.
  arXiv  Detail & Related papers  (2024-04-23T18:41:56Z)
• Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models [66.05234562835136]
  We present MuDI, a novel framework that enables multi-subject personalization. Our main idea is to utilize segmented subjects generated by a foundation model for segmentation. Experimental results show that our MuDI can produce high-quality personalized images without identity mixing.
  arXiv  Detail & Related papers  (2024-04-05T17:45:22Z)
• IDAdapter: Learning Mixed Features for Tuning-Free Personalization of Text-to-Image Models [31.762112403595612]
  IDAdapter is a tuning-free approach that enhances the diversity and identity preservation in personalized image generation from a single face image. During the training phase, we incorporate mixed features from multiple reference images of a specific identity to enrich identity-related content details.
  arXiv  Detail & Related papers  (2024-03-20T12:13:04Z)
• Infinite-ID: Identity-preserved Personalization via ID-semantics Decoupling Paradigm [31.06269858216316]
  We propose Infinite-ID, an ID-semantics decoupling paradigm for identity-preserved personalization. We introduce an identity-enhanced training, incorporating an additional image cross-attention module to capture sufficient ID information. We also introduce a feature interaction mechanism that combines a mixed attention module with an AdaIN-mean operation to seamlessly merge the two streams.
  arXiv  Detail & Related papers  (2024-03-18T13:39:53Z)
• MuLan: Multimodal-LLM Agent for Progressive and Interactive Multi-Object Diffusion [81.7514869897233]
  We develop a training-free Multimodal-LLM agent (MuLan), as a human painter, that can progressively generate multi-object. MuLan harnesses a large language model (LLM) to decompose a prompt to a sequence of sub-tasks, each generating only one object by stable diffusion. MuLan also adopts a vision-language model (VLM) to provide feedback to the image generated in each sub-task and control the diffusion model to re-generate the image if it violates the original prompt.
  arXiv  Detail & Related papers  (2024-02-20T06:14:30Z)
• InstantID: Zero-shot Identity-Preserving Generation in Seconds [21.04236321562671]
  We introduce InstantID, a powerful diffusion model-based solution for ID embedding. Our plug-and-play module adeptly handles image personalization in various styles using just a single facial image. Our work seamlessly integrates with popular pre-trained text-to-image diffusion models like SD1.5 and SDXL.
  arXiv  Detail & Related papers  (2024-01-15T07:50:18Z)
• PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding [102.07914175196817]
  PhotoMaker is an efficient personalized text-to-image generation method. It encodes an arbitrary number of input ID images into a stack ID embedding for preserving ID information.
  arXiv  Detail & Related papers  (2023-12-07T17:32:29Z)
• Identity Encoder for Personalized Diffusion [57.1198884486401]
  We propose an encoder-based approach for personalization. We learn an identity encoder which can extract an identity representation from a set of reference images of a subject. We show that our approach consistently outperforms existing fine-tuning based approach in both image generation and reconstruction.
  arXiv  Detail & Related papers  (2023-04-14T23:32:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.