InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning

• URL: http://arxiv.org/abs/2304.03411v1
• Date: Thu, 6 Apr 2023 23:26:38 GMT
• Title: InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning
• Authors: Jing Shi, Wei Xiong, Zhe Lin, Hyun Joon Jung
• Abstract summary: We propose InstantBooth, a novel approach built upon pre-trained text-to-image models. Our model can generate competitive results on unseen concepts concerning language-image alignment, image fidelity, and identity preservation.
• Score: 20.127745565621616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in personalized image generation allow a pre-trained text-to-image model to learn a new concept from a set of images. However, existing personalization approaches usually require heavy test-time finetuning for each concept, which is time-consuming and difficult to scale. We propose InstantBooth, a novel approach built upon pre-trained text-to-image models that enables instant text-guided image personalization without any test-time finetuning. We achieve this with several major components. First, we learn the general concept of the input images by converting them to a textual token with a learnable image encoder. Second, to keep the fine details of the identity, we learn rich visual feature representation by introducing a few adapter layers to the pre-trained model. We train our components only on text-image pairs without using paired images of the same concept. Compared to test-time finetuning-based methods like DreamBooth and Textual-Inversion, our model can generate competitive results on unseen concepts concerning language-image alignment, image fidelity, and identity preservation while being 100 times faster.

Related papers

• JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation [49.997839600988875]
  Existing personalization methods rely on finetuning a text-to-image foundation model on a user's custom dataset. We propose Joint-Image Diffusion (jedi), an effective technique for learning a finetuning-free personalization model. Our model achieves state-of-the-art generation quality, both quantitatively and qualitatively, significantly outperforming both the prior finetuning-based and finetuning-free personalization baselines.
  arXiv  Detail & Related papers  (2024-07-08T17:59:02Z)
• FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition [49.2208591663092]
  FreeCustom is a tuning-free method to generate customized images of multi-concept composition based on reference concepts. We introduce a new multi-reference self-attention (MRSA) mechanism and a weighted mask strategy. Our method outperforms or performs on par with other training-based methods in terms of multi-concept composition and single-concept customization.
  arXiv  Detail & Related papers  (2024-05-22T17:53:38Z)
• Fast Personalized Text-to-Image Syntheses With Attention Injection [17.587109812987475]
  We propose an effective and fast approach that could balance the text-image consistency and identity consistency of the generated image and reference image. Our method can generate personalized images without any fine-tuning while maintaining the inherent text-to-image generation ability of diffusion models.
  arXiv  Detail & Related papers  (2024-03-17T17:42:02Z)
• InstructBooth: Instruction-following Personalized Text-to-Image Generation [30.89054609185801]
  InstructBooth is a novel method designed to enhance image-text alignment in personalized text-to-image models. Our approach first personalizes text-to-image models with a small number of subject-specific images using a unique identifier. After personalization, we fine-tune personalized text-to-image models using reinforcement learning to maximize a reward that quantifies image-text alignment.
  arXiv  Detail & Related papers  (2023-12-04T20:34:46Z)
• Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models [59.094601993993535]
  Text-to-image (T2I) personalization allows users to combine their own visual concepts in natural language prompts. Most existing encoders are limited to a single-class domain, which hinders their ability to handle diverse concepts. We propose a domain-agnostic method that does not require any specialized dataset or prior information about the personalized concepts.
  arXiv  Detail & Related papers  (2023-07-13T17:46:42Z)
• Break-A-Scene: Extracting Multiple Concepts from a Single Image [80.47666266017207]
  We introduce the task of textual scene decomposition. We propose augmenting the input image with masks that indicate the presence of target concepts. We then present a novel two-phase customization process.
  arXiv  Detail & Related papers  (2023-05-25T17:59:04Z)
• Designing an Encoder for Fast Personalization of Text-to-Image Models [57.62449900121022]
  We propose an encoder-based domain-tuning approach for text-to-image personalization. We employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively ingest additional concepts.
  arXiv  Detail & Related papers  (2023-02-23T18:46:41Z)
• Multi-Concept Customization of Text-to-Image Diffusion [51.8642043743222]
  We propose Custom Diffusion, an efficient method for augmenting existing text-to-image models. We find that only optimizing a few parameters in the text-to-image conditioning mechanism is sufficiently powerful to represent new concepts. Our model generates variations of multiple new concepts and seamlessly composes them with existing concepts in novel settings.
  arXiv  Detail & Related papers  (2022-12-08T18:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.