Zero-Shot Personalization of Objects via Textual Inversion

• URL: http://arxiv.org/abs/2603.23010v1
• Date: Tue, 24 Mar 2026 09:54:30 GMT
• Title: Zero-Shot Personalization of Objects via Textual Inversion
• Abstract summary: We propose a novel framework that employs a learned network to predict object-specific textual embeddings.<n>This design enables rapid, zero-shot personalization of a wide range of objects in a single forward pass.<n>To the best of our knowledge, this work represents the first attempt to achieve such general-purpose, training-free personalization within diffusion models.
• Score: 36.82648022347006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in text-to-image diffusion models have substantially improved the quality of image customization, enabling the synthesis of highly realistic images. Despite this progress, achieving fast and efficient personalization remains a key challenge, particularly for real-world applications. Existing approaches primarily accelerate customization for human subjects by injecting identity-specific embeddings into diffusion models, but these strategies do not generalize well to arbitrary object categories, limiting their applicability. To address this limitation, we propose a novel framework that employs a learned network to predict object-specific textual inversion embeddings, which are subsequently integrated into the UNet timesteps of a diffusion model for text-conditional customization. This design enables rapid, zero-shot personalization of a wide range of objects in a single forward pass, offering both flexibility and scalability. Extensive experiments across multiple tasks and settings demonstrate the effectiveness of our approach, highlighting its potential to support fast, versatile, and inclusive image customization. To the best of our knowledge, this work represents the first attempt to achieve such general-purpose, training-free personalization within diffusion models, paving the way for future research in personalized image generation.

Related papers

• Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models [20.582222123619285]
  We propose a training-free framework that formulates personalized content editing as the optimization of edited images in the latent space.<n>A coarse-to-fine strategy is proposed that employs text energy guidance at the early stage to achieve a natural transition toward the target class.<n>Our method excels in object replacement even with a large domain gap.
  arXiv  Detail & Related papers  (2025-03-06T08:52:29Z)
• InsertDiffusion: Identity Preserving Visualization of Objects through a Training-Free Diffusion Architecture [0.0]
  InsertDiffusion is a training-free diffusion architecture that efficiently embeds objects into images. Our approach utilizes off-the-shelf generative models and eliminates the need for fine-tuning. By decomposing the generation task into independent steps, InsertDiffusion offers a scalable solution.
  arXiv  Detail & Related papers  (2024-07-15T10:15:58Z)
• Powerful and Flexible: Personalized Text-to-Image Generation via Reinforcement Learning [40.06403155373455]
  We propose a novel reinforcement learning framework for personalized text-to-image generation. Our proposed approach outperforms existing state-of-the-art methods by a large margin on visual fidelity while maintaining text-alignment.
  arXiv  Detail & Related papers  (2024-07-09T08:11:53Z)
• JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation [49.997839600988875]
  Existing personalization methods rely on finetuning a text-to-image foundation model on a user's custom dataset. We propose Joint-Image Diffusion (jedi), an effective technique for learning a finetuning-free personalization model. Our model achieves state-of-the-art generation quality, both quantitatively and qualitatively, significantly outperforming both the prior finetuning-based and finetuning-free personalization baselines.
  arXiv  Detail & Related papers  (2024-07-08T17:59:02Z)
• FreeCompose: Generic Zero-Shot Image Composition with Diffusion Prior [50.0535198082903]
  We offer a novel approach to image composition, which integrates multiple input images into a single, coherent image. We showcase the potential of utilizing the powerful generative prior inherent in large-scale pre-trained diffusion models to accomplish generic image composition.
  arXiv  Detail & Related papers  (2024-07-06T03:35:43Z)
• Training-Free Consistent Text-to-Image Generation [80.4814768762066]
  Text-to-image models can portray the same subject across diverse prompts. Existing approaches fine-tune the model to teach it new words that describe specific user-provided subjects. We present ConsiStory, a training-free approach that enables consistent subject generation by sharing the internal activations of the pretrained model.
  arXiv  Detail & Related papers  (2024-02-05T18:42:34Z)
• Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models [59.094601993993535]
  Text-to-image (T2I) personalization allows users to combine their own visual concepts in natural language prompts. Most existing encoders are limited to a single-class domain, which hinders their ability to handle diverse concepts. We propose a domain-agnostic method that does not require any specialized dataset or prior information about the personalized concepts.
  arXiv  Detail & Related papers  (2023-07-13T17:46:42Z)
• Generate Anything Anywhere in Any Scene [25.75076439397536]
  We propose a controllable text-to-image diffusion model for personalized object generation. Our approach demonstrates significant potential for various applications, such as those in art, entertainment, and advertising design.
  arXiv  Detail & Related papers  (2023-06-29T17:55:14Z)
• Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Models [55.04969603431266]
  This paper proposes a method for generating images of customized objects specified by users. The method is based on a general framework that bypasses the lengthy optimization required by previous approaches. We demonstrate through experiments that our proposed method is able to synthesize images with compelling output quality, appearance diversity, and object fidelity.
  arXiv  Detail & Related papers  (2023-04-05T17:59:32Z)
• Designing an Encoder for Fast Personalization of Text-to-Image Models [57.62449900121022]
  We propose an encoder-based domain-tuning approach for text-to-image personalization. We employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively ingest additional concepts.
  arXiv  Detail & Related papers  (2023-02-23T18:46:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.