InstantCharacter: Personalize Any Characters with a Scalable Diffusion Transformer Framework

• URL: http://arxiv.org/abs/2504.12395v1
• Date: Wed, 16 Apr 2025 18:01:59 GMT
• Title: InstantCharacter: Personalize Any Characters with a Scalable Diffusion Transformer Framework
• Authors: Jiale Tao, Yanbing Zhang, Qixun Wang, Yiji Cheng, Haofan Wang, Xu Bai, Zhengguang Zhou, Ruihuang Li, Linqing Wang, Chunyu Wang, Qin Lin, Qinglin Lu,
• Abstract summary: InstantCharacter is a scalable framework for character customization built upon a foundation diffusion transformer.<n>It achieves open-domain personalization across diverse character appearances, poses, and styles while maintaining high-fidelity results.
• Score: 24.29397138274732
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current learning-based subject customization approaches, predominantly relying on U-Net architectures, suffer from limited generalization ability and compromised image quality. Meanwhile, optimization-based methods require subject-specific fine-tuning, which inevitably degrades textual controllability. To address these challenges, we propose InstantCharacter, a scalable framework for character customization built upon a foundation diffusion transformer. InstantCharacter demonstrates three fundamental advantages: first, it achieves open-domain personalization across diverse character appearances, poses, and styles while maintaining high-fidelity results. Second, the framework introduces a scalable adapter with stacked transformer encoders, which effectively processes open-domain character features and seamlessly interacts with the latent space of modern diffusion transformers. Third, to effectively train the framework, we construct a large-scale character dataset containing 10-million-level samples. The dataset is systematically organized into paired (multi-view character) and unpaired (text-image combinations) subsets. This dual-data structure enables simultaneous optimization of identity consistency and textual editability through distinct learning pathways. Qualitative experiments demonstrate the advanced capabilities of InstantCharacter in generating high-fidelity, text-controllable, and character-consistent images, setting a new benchmark for character-driven image generation. Our source code is available at https://github.com/Tencent/InstantCharacter.

Related papers

• Identity-Preserving Text-to-Image Generation via Dual-Level Feature Decoupling and Expert-Guided Fusion [35.67333978414322]
  We propose a novel framework that improves the separation of identity-related and identity-unrelated features.<n>Our framework consists of two key components: an Implicit-Explicit foreground-background Decoupling Module and a Feature Fusion Module.
  arXiv  Detail & Related papers  (2025-05-28T13:40:46Z)
• GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing [23.64662356622401]
  We present GlyphMastero, a specialized glyph encoder designed to guide the latent diffusion model for generating texts with stroke-level precision.<n>Our method achieves an 18.02% improvement in sentence accuracy over the state-of-the-art scene text editing baseline.
  arXiv  Detail & Related papers  (2025-05-08T03:11:58Z)
• DisEnvisioner: Disentangled and Enriched Visual Prompt for Customized Image Generation [22.599542105037443]
  DisEnvisioner is a novel approach for effectively extracting and enriching the subject-essential features while filtering out -irrelevant information. Specifically, the feature of the subject and other irrelevant components are effectively separated into distinctive visual tokens, enabling a much more accurate customization. Experiments demonstrate the superiority of our approach over existing methods in instruction response (editability), ID consistency, inference speed, and the overall image quality.
  arXiv  Detail & Related papers  (2024-10-02T22:29:14Z)
• Fusion is all you need: Face Fusion for Customized Identity-Preserving Image Synthesis [7.099258248662009]
  Text-to-image (T2I) models have significantly advanced the development of artificial intelligence. However, existing T2I-based methods often struggle to accurately reproduce the appearance of individuals from a reference image. We leverage the pre-trained UNet from Stable Diffusion to incorporate the target face image directly into the generation process.
  arXiv  Detail & Related papers  (2024-09-27T19:31:04Z)
• STAR: Scale-wise Text-conditioned AutoRegressive image generation [38.98271279816512]
  We introduce STAR, a text-to-image model that employs a scale-wise auto-regressive paradigm.<n> STAR enables text-driven image generation up to 1024$times$1024 through three key designs.
  arXiv  Detail & Related papers  (2024-06-16T03:45:45Z)
• Language Guided Domain Generalized Medical Image Segmentation [68.93124785575739]
  Single source domain generalization holds promise for more reliable and consistent image segmentation across real-world clinical settings. We propose an approach that explicitly leverages textual information by incorporating a contrastive learning mechanism guided by the text encoder features. Our approach achieves favorable performance against existing methods in literature.
  arXiv  Detail & Related papers  (2024-04-01T17:48:15Z)
• Masked Generative Story Transformer with Character Guidance and Caption Augmentation [2.1392064955842023]
  Story visualization is a challenging generative vision task, that requires both visual quality and consistency between different frames in generated image sequences. Previous approaches either employ some kind of memory mechanism to maintain context throughout an auto-regressive generation of the image sequence, or model the generation of the characters and their background separately. We propose a completely parallel transformer-based approach, relying on Cross-Attention with past and future captions to achieve consistency.
  arXiv  Detail & Related papers  (2024-03-13T13:10:20Z)
• Disentangled Representation Learning for Controllable Person Image Generation [29.719070087384512]
  We propose a novel framework named DRL-CPG to learn disentangled latent representation for controllable person image generation. To our knowledge, we are the first to learn disentangled latent representations with transformers for person image generation.
  arXiv  Detail & Related papers  (2023-12-10T07:15:58Z)
• PV2TEA: Patching Visual Modality to Textual-Established Information Extraction [59.76117533540496]
  We patch the visual modality to the textual-established attribute information extractor. PV2TEA is an encoder-decoder architecture equipped with three bias reduction schemes. Empirical results on real-world e-Commerce datasets demonstrate up to 11.74% absolute (20.97% relatively) F1 increase over unimodal baselines.
  arXiv  Detail & Related papers  (2023-06-01T05:39:45Z)
• DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation [56.514462874501675]
  We propose a dynamic sparse attention based Transformer model to achieve fine-level matching with favorable efficiency. The heart of our approach is a novel dynamic-attention unit, dedicated to covering the variation on the optimal number of tokens one position should focus on. Experiments on three applications, pose-guided person image generation, edge-based face synthesis, and undistorted image style transfer, demonstrate that DynaST achieves superior performance in local details.
  arXiv  Detail & Related papers  (2022-07-13T11:12:03Z)
• Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors [58.71128866226768]
  Recent text-to-image generation methods have incrementally improved the generated image fidelity and text relevancy. We propose a novel text-to-image method that addresses these gaps by (i) enabling a simple control mechanism complementary to text in the form of a scene. Our model achieves state-of-the-art FID and human evaluation results, unlocking the ability to generate high fidelity images in a resolution of 512x512 pixels.
  arXiv  Detail & Related papers  (2022-03-24T15:44:50Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)
• Generating Person Images with Appearance-aware Pose Stylizer [66.44220388377596]
  We present a novel end-to-end framework to generate realistic person images based on given person poses and appearances. The core of our framework is a novel generator called Appearance-aware Pose Stylizer (APS) which generates human images by coupling the target pose with the conditioned person appearance progressively.
  arXiv  Detail & Related papers  (2020-07-17T15:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.