FreeTuner: Any Subject in Any Style with Training-free Diffusion

• URL: http://arxiv.org/abs/2405.14201v2
• Date: Mon, 27 May 2024 03:14:02 GMT
• Title: FreeTuner: Any Subject in Any Style with Training-free Diffusion
• Authors: Youcan Xu, Zhen Wang, Jun Xiao, Wei Liu, Long Chen,
• Abstract summary: FreeTuner is a flexible and training-free method for compositional personalization that can generate any user-provided subject in any user-provided style. Our approach employs a disentanglement strategy that separates the generation process into two stages to effectively mitigate concept entanglement.
• Score: 17.18034002758044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the advance of diffusion models, various personalized image generation methods have been proposed. However, almost all existing work only focuses on either subject-driven or style-driven personalization. Meanwhile, state-of-the-art methods face several challenges in realizing compositional personalization, i.e., composing different subject and style concepts, such as concept disentanglement, unified reconstruction paradigm, and insufficient training data. To address these issues, we introduce FreeTuner, a flexible and training-free method for compositional personalization that can generate any user-provided subject in any user-provided style (see Figure 1). Our approach employs a disentanglement strategy that separates the generation process into two stages to effectively mitigate concept entanglement. FreeTuner leverages the intermediate features within the diffusion model for subject concept representation and introduces style guidance to align the synthesized images with the style concept, ensuring the preservation of both the subject's structure and the style's aesthetic features. Extensive experiments have demonstrated the generation ability of FreeTuner across various personalization settings.

Related papers

• Personalize Anything for Free with Diffusion Transformer [20.385520869825413]
  Recent training-free approaches struggle with identity preservation, applicability, and compatibility with diffusion transformers (DiTs) We uncover the untapped potential of DiT, where simply replacing denoising tokens with those of a reference subject achieves zero-shot subject reconstruction. We propose textbfPersonalize Anything, a training-free framework that achieves personalized image generation in DiT through: 1) timestep-adaptive token replacement that enforces subject consistency via early-stage injection and enhances flexibility through late-stage regularization, and 2) patch perturbation strategies to boost structural diversity.
  arXiv  Detail & Related papers  (2025-03-16T17:51:16Z)
• SubZero: Composing Subject, Style, and Action via Zero-Shot Personalization [46.75550543879637]
  Diffusion models are increasingly popular for generative tasks, including personalized composition of subjects and styles. SubZero is a novel framework to generate any subject in any style, performing any action without the need for fine-tuning. We show that our proposed approach, while suitable for running on-edge, shows significant improvements over state-of-the-art works performing subject, style and action composition.
  arXiv  Detail & Related papers  (2025-02-27T01:33:28Z)
• FlipConcept: Tuning-Free Multi-Concept Personalization for Text-to-Image Generation [0.0]
  Methods that integrate multiple personalized concepts into a single image have garnered significant attention in the field of text-to-image (T2I) generation. Existing methods experience performance degradation in complex scenes with multiple objects due to distortions in non-personalized regions. We propose FlipConcept, a novel approach that seamlessly integrates multiple personalized concepts into a single image without requiring additional tuning.
  arXiv  Detail & Related papers  (2025-02-21T04:37:18Z)
• Personalized Image Generation with Deep Generative Models: A Decade Survey [51.26287478042516]
  We present a review of generalized personalized image generation across various generative models. We first define a unified framework that standardizes the personalization process across different generative models. We then provide an in-depth analysis of personalization techniques within each generative model, highlighting their unique contributions and innovations.
  arXiv  Detail & Related papers  (2025-02-18T17:34:04Z)
• Imagine yourself: Tuning-Free Personalized Image Generation [39.63411174712078]
  We introduce Imagine yourself, a state-of-the-art model designed for personalized image generation. It operates as a tuning-free model, enabling all users to leverage a shared framework without individualized adjustments. Our study demonstrates that Imagine yourself surpasses the state-of-the-art personalization model, exhibiting superior capabilities in identity preservation, visual quality, and text alignment.
  arXiv  Detail & Related papers  (2024-09-20T09:21:49Z)
• Concept Conductor: Orchestrating Multiple Personalized Concepts in Text-to-Image Synthesis [14.21719970175159]
  Concept Conductor is designed to ensure visual fidelity and correct layout in multi-concept customization. We present a concept injection technique that employs shape-aware masks to specify the generation area for each concept. Our method supports the combination of any number of concepts and maintains high fidelity even when dealing with visually similar concepts.
  arXiv  Detail & Related papers  (2024-08-07T08:43:58Z)
• JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation [49.997839600988875]
  Existing personalization methods rely on finetuning a text-to-image foundation model on a user's custom dataset. We propose Joint-Image Diffusion (jedi), an effective technique for learning a finetuning-free personalization model. Our model achieves state-of-the-art generation quality, both quantitatively and qualitatively, significantly outperforming both the prior finetuning-based and finetuning-free personalization baselines.
  arXiv  Detail & Related papers  (2024-07-08T17:59:02Z)
• Ada-adapter:Fast Few-shot Style Personlization of Diffusion Model with Pre-trained Image Encoder [57.574544285878794]
  Ada-Adapter is a novel framework for few-shot style personalization of diffusion models. Our method enables efficient zero-shot style transfer utilizing a single reference image. We demonstrate the effectiveness of our approach on various artistic styles, including flat art, 3D rendering, and logo design.
  arXiv  Detail & Related papers  (2024-07-08T02:00:17Z)
• Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models [85.14042557052352]
  We introduce Concept Weaver, a method for composing customized text-to-image diffusion models at inference time. We show that Concept Weaver can generate multiple custom concepts with higher identity fidelity compared to alternative approaches.
  arXiv  Detail & Related papers  (2024-04-05T06:41:27Z)
• Training-Free Consistent Text-to-Image Generation [80.4814768762066]
  Text-to-image models can portray the same subject across diverse prompts. Existing approaches fine-tune the model to teach it new words that describe specific user-provided subjects. We present ConsiStory, a training-free approach that enables consistent subject generation by sharing the internal activations of the pretrained model.
  arXiv  Detail & Related papers  (2024-02-05T18:42:34Z)
• Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization [56.12990759116612]
  Pick-and-Draw is a training-free semantic guidance approach to boost identity consistency and generative diversity for personalization methods. The proposed approach can be applied to any personalized diffusion models and requires as few as a single reference image.
  arXiv  Detail & Related papers  (2024-01-30T05:56:12Z)
• PALP: Prompt Aligned Personalization of Text-to-Image Models [68.91005384187348]
  Existing personalization methods compromise personalization ability or the alignment to complex prompts. We propose a new approach focusing on personalization methods for a emphsingle prompt to address this issue. Our method excels in improving text alignment, enabling the creation of images with complex and intricate prompts.
  arXiv  Detail & Related papers  (2024-01-11T18:35:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.