CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models

• URL: http://arxiv.org/abs/2404.15677v2
• Date: Sat, 27 Apr 2024 14:24:15 GMT
• Title: CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models
• Authors: Qinghe Wang, Baolu Li, Xiaomin Li, Bing Cao, Liqian Ma, Huchuan Lu, Xu Jia,
• Abstract summary: CharacterFactory is a framework that allows sampling new characters with consistent identities in the latent space of GANs. The whole model only takes 10 minutes for training, and can sample infinite characters end-to-end during inference.
• Score: 58.37569942713456
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in text-to-image models have opened new frontiers in human-centric generation. However, these models cannot be directly employed to generate images with consistent newly coined identities. In this work, we propose CharacterFactory, a framework that allows sampling new characters with consistent identities in the latent space of GANs for diffusion models. More specifically, we consider the word embeddings of celeb names as ground truths for the identity-consistent generation task and train a GAN model to learn the mapping from a latent space to the celeb embedding space. In addition, we design a context-consistent loss to ensure that the generated identity embeddings can produce identity-consistent images in various contexts. Remarkably, the whole model only takes 10 minutes for training, and can sample infinite characters end-to-end during inference. Extensive experiments demonstrate excellent performance of the proposed CharacterFactory on character creation in terms of identity consistency and editability. Furthermore, the generated characters can be seamlessly combined with the off-the-shelf image/video/3D diffusion models. We believe that the proposed CharacterFactory is an important step for identity-consistent character generation. Project page is available at: https://qinghew.github.io/CharacterFactory/.

Related papers

• Storynizor: Consistent Story Generation via Inter-Frame Synchronized and Shuffled ID Injection [27.412361280397057]
  We introduce Storynizor, a model capable of generating coherent stories with strong inter-frame character consistency. Key innovation of Storynizor lies in its key modules: ID-Synchronizer and ID-Injector. To facilitate the training of Storynizor, we have curated a novel dataset called StoryDB comprising 100, 000 images.
  arXiv  Detail & Related papers  (2024-09-29T09:15:51Z)
• StoryMaker: Towards Holistic Consistent Characters in Text-to-image Generation [10.652011707000202]
  We introduce StoryMaker, a personalization solution that preserves not only facial consistency but also clothing, hairstyles, and body consistency. StoryMaker supports numerous applications and is compatible with other societal plug-ins.
  arXiv  Detail & Related papers  (2024-09-19T08:53:06Z)
• Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Models [79.21968152209193]
  We introduce the NewEpisode benchmark to evaluate generative models' adaptability in generating new stories with fresh characters. We propose EpicEvo, a method that customizes a diffusion-based visual story generation model with a single story featuring the new characters seamlessly integrating them into established character dynamics.
  arXiv  Detail & Related papers  (2024-05-20T07:54:03Z)
• Arc2Face: A Foundation Model for ID-Consistent Human Faces [95.00331107591859]
  Arc2Face is an identity-conditioned face foundation model. It can generate diverse photo-realistic images with an unparalleled degree of face similarity than existing models.
  arXiv  Detail & Related papers  (2024-03-18T10:32:51Z)
• Beyond Inserting: Learning Identity Embedding for Semantic-Fidelity Personalized Diffusion Generation [21.739328335601716]
  This paper focuses on inserting accurate and interactive ID embedding into the Stable Diffusion Model for personalized generation. We propose a face-wise attention loss to fit the face region instead of entangling ID-unrelated information, such as face layout and background. Our results exhibit superior ID accuracy, text-based manipulation ability, and generalization compared to previous methods.
  arXiv  Detail & Related papers  (2024-01-31T11:52:33Z)
• StableIdentity: Inserting Anybody into Anywhere at First Sight [57.99693188913382]
  We propose StableIdentity, which allows identity-consistent recontextualization with just one face image. We are the first to directly inject the identity learned from a single image into video/3D generation without finetuning.
  arXiv  Detail & Related papers  (2024-01-29T09:06:15Z)
• StoryGPT-V: Large Language Models as Consistent Story Visualizers [39.790319429455856]
  generative models have demonstrated impressive capabilities in generating realistic and visually pleasing images grounded on textual prompts. Yet, the emerging Large Language Model (LLM) showcases robust reasoning abilities to navigate through ambiguous references. We introduce textbfStoryGPT-V, which leverages the merits of the latent diffusion (LDM) and LLM to produce images with consistent and high-quality characters.
  arXiv  Detail & Related papers  (2023-12-04T18:14:29Z)
• When StyleGAN Meets Stable Diffusion: a $\mathscr{W}_+$ Adapter for Personalized Image Generation [60.305112612629465]
  Text-to-image diffusion models have excelled in producing diverse, high-quality, and photo-realistic images. We present a novel use of the extended StyleGAN embedding space $mathcalW_+$ to achieve enhanced identity preservation and disentanglement for diffusion models. Our method adeptly generates personalized text-to-image outputs that are not only compatible with prompt descriptions but also amenable to common StyleGAN editing directions.
  arXiv  Detail & Related papers  (2023-11-29T09:05:14Z)
• eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers [87.52504764677226]
  Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. We train an ensemble of text-to-image diffusion models specialized for different stages synthesis. Our ensemble of diffusion models, called eDiffi, results in improved text alignment while maintaining the same inference cost.
  arXiv  Detail & Related papers  (2022-11-02T17:43:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.