SubZero: Composing Subject, Style, and Action via Zero-Shot Personalization

• URL: http://arxiv.org/abs/2502.19673v1
• Date: Thu, 27 Feb 2025 01:33:28 GMT
• Title: SubZero: Composing Subject, Style, and Action via Zero-Shot Personalization
• Authors: Shubhankar Borse, Kartikeya Bhardwaj, Mohammad Reza Karimi Dastjerdi, Hyojin Park, Shreya Kadambi, Shobitha Shivakumar, Prathamesh Mandke, Ankita Nayak, Harris Teague, Munawar Hayat, Fatih Porikli,
• Abstract summary: Diffusion models are increasingly popular for generative tasks, including personalized composition of subjects and styles.<n>SubZero is a novel framework to generate any subject in any style, performing any action without the need for fine-tuning.<n>We show that our proposed approach, while suitable for running on-edge, shows significant improvements over state-of-the-art works performing subject, style and action composition.
• Score: 46.75550543879637
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models are increasingly popular for generative tasks, including personalized composition of subjects and styles. While diffusion models can generate user-specified subjects performing text-guided actions in custom styles, they require fine-tuning and are not feasible for personalization on mobile devices. Hence, tuning-free personalization methods such as IP-Adapters have progressively gained traction. However, for the composition of subjects and styles, these works are less flexible due to their reliance on ControlNet, or show content and style leakage artifacts. To tackle these, we present SubZero, a novel framework to generate any subject in any style, performing any action without the need for fine-tuning. We propose a novel set of constraints to enhance subject and style similarity, while reducing leakage. Additionally, we propose an orthogonalized temporal aggregation scheme in the cross-attention blocks of denoising model, effectively conditioning on a text prompt along with single subject and style images. We also propose a novel method to train customized content and style projectors to reduce content and style leakage. Through extensive experiments, we show that our proposed approach, while suitable for running on-edge, shows significant improvements over state-of-the-art works performing subject, style and action composition.

Related papers

• StyleStudio: Text-Driven Style Transfer with Selective Control of Style Elements [10.752464085587267]
  Text-driven style transfer aims to merge the style of a reference image with content described by a text prompt. Recent advancements in text-to-image models have improved the transformations of nuance style, yet significant challenges remain. We propose three complementary strategies to address these issues.
  arXiv  Detail & Related papers  (2024-12-11T16:13:23Z)
• DiffArtist: Towards Aesthetic-Aligned Diffusion Model Control for Training-free Text-Driven Stylization [19.5597806965592]
  Diffusion models entangle content and style generation during the denoising process.<n>DiffusionArtist is the first approach that enables aesthetic-aligned control of content and style during the entire diffusion process.
  arXiv  Detail & Related papers  (2024-07-22T17:58:05Z)
• InstantStyle-Plus: Style Transfer with Content-Preserving in Text-to-Image Generation [4.1177497612346]
  Style transfer is an inventive process designed to create an image that maintains the essence of the original while embracing the visual style of another. We introduce InstantStyle-Plus, an approach that prioritizes the integrity of the original content while seamlessly integrating the target style.
  arXiv  Detail & Related papers  (2024-06-30T18:05:33Z)
• ArtWeaver: Advanced Dynamic Style Integration via Diffusion Model [73.95608242322949]
  Stylized Text-to-Image Generation (STIG) aims to generate images from text prompts and style reference images. We present ArtWeaver, a novel framework that leverages pretrained Stable Diffusion to address challenges such as misinterpreted styles and inconsistent semantics.
  arXiv  Detail & Related papers  (2024-05-24T07:19:40Z)
• FreeTuner: Any Subject in Any Style with Training-free Diffusion [17.18034002758044]
  FreeTuner is a flexible and training-free method for compositional personalization that can generate any user-provided subject in any user-provided style. Our approach employs a disentanglement strategy that separates the generation process into two stages to effectively mitigate concept entanglement.
  arXiv  Detail & Related papers  (2024-05-23T06:01:13Z)
• InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation [5.364489068722223]
  The concept of style is inherently underdetermined, encompassing a multitude of elements such as color, material, atmosphere, design, and structure. Inversion-based methods are prone to style degradation, often resulting in the loss of fine-grained details. adapter-based approaches frequently require meticulous weight tuning for each reference image to achieve a balance between style intensity and text controllability.
  arXiv  Detail & Related papers  (2024-04-03T13:34:09Z)
• PALP: Prompt Aligned Personalization of Text-to-Image Models [68.91005384187348]
  Existing personalization methods compromise personalization ability or the alignment to complex prompts. We propose a new approach focusing on personalization methods for a emphsingle prompt to address this issue. Our method excels in improving text alignment, enabling the creation of images with complex and intricate prompts.
  arXiv  Detail & Related papers  (2024-01-11T18:35:33Z)
• Style Aligned Image Generation via Shared Attention [61.121465570763085]
  We introduce StyleAligned, a technique designed to establish style alignment among a series of generated images. By employing minimal attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. Our method's evaluation across diverse styles and text prompts demonstrates high-quality and fidelity.
  arXiv  Detail & Related papers  (2023-12-04T18:55:35Z)
• StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter [78.75422651890776]
  StyleCrafter is a generic method that enhances pre-trained T2V models with a style control adapter. To promote content-style disentanglement, we remove style descriptions from the text prompt and extract style information solely from the reference image. StyleCrafter efficiently generates high-quality stylized videos that align with the content of the texts and resemble the style of the reference images.
  arXiv  Detail & Related papers  (2023-12-01T03:53:21Z)
• StyleAdapter: A Unified Stylized Image Generation Model [97.24936247688824]
  StyleAdapter is a unified stylized image generation model capable of producing a variety of stylized images. It can be integrated with existing controllable synthesis methods, such as T2I-adapter and ControlNet.
  arXiv  Detail & Related papers  (2023-09-04T19:16:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.