InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation

• URL: http://arxiv.org/abs/2404.02733v2
• Date: Thu, 4 Apr 2024 19:42:32 GMT
• Title: InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation
• Authors: Haofan Wang, Matteo Spinelli, Qixun Wang, Xu Bai, Zekui Qin, Anthony Chen,
• Abstract summary: The concept of style is inherently underdetermined, encompassing a multitude of elements such as color, material, atmosphere, design, and structure. Inversion-based methods are prone to style degradation, often resulting in the loss of fine-grained details. adapter-based approaches frequently require meticulous weight tuning for each reference image to achieve a balance between style intensity and text controllability.
• Score: 5.364489068722223
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tuning-free diffusion-based models have demonstrated significant potential in the realm of image personalization and customization. However, despite this notable progress, current models continue to grapple with several complex challenges in producing style-consistent image generation. Firstly, the concept of style is inherently underdetermined, encompassing a multitude of elements such as color, material, atmosphere, design, and structure, among others. Secondly, inversion-based methods are prone to style degradation, often resulting in the loss of fine-grained details. Lastly, adapter-based approaches frequently require meticulous weight tuning for each reference image to achieve a balance between style intensity and text controllability. In this paper, we commence by examining several compelling yet frequently overlooked observations. We then proceed to introduce InstantStyle, a framework designed to address these issues through the implementation of two key strategies: 1) A straightforward mechanism that decouples style and content from reference images within the feature space, predicated on the assumption that features within the same space can be either added to or subtracted from one another. 2) The injection of reference image features exclusively into style-specific blocks, thereby preventing style leaks and eschewing the need for cumbersome weight tuning, which often characterizes more parameter-heavy designs.Our work demonstrates superior visual stylization outcomes, striking an optimal balance between the intensity of style and the controllability of textual elements. Our codes will be available at https://github.com/InstantStyle/InstantStyle.

Related papers

• Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics [3.9717825324709413]
  Style has been primarily considered in terms of artistic elements such as colors, brushstrokes, and lighting. In this study, we propose a zero-shot scheme for image variation with coordinated semantics.
  arXiv  Detail & Related papers  (2024-10-24T08:34:57Z)
• ZePo: Zero-Shot Portrait Stylization with Faster Sampling [61.14140480095604]
  This paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control.
  arXiv  Detail & Related papers  (2024-08-10T08:53:41Z)
• InstantStyle-Plus: Style Transfer with Content-Preserving in Text-to-Image Generation [4.1177497612346]
  Style transfer is an inventive process designed to create an image that maintains the essence of the original while embracing the visual style of another. We introduce InstantStyle-Plus, an approach that prioritizes the integrity of the original content while seamlessly integrating the target style.
  arXiv  Detail & Related papers  (2024-06-30T18:05:33Z)
• Visual Style Prompting with Swapping Self-Attention [26.511518230332758]
  We propose a novel approach to produce a diverse range of images while maintaining specific style elements and nuances. During the denoising process, we keep the query from original features while swapping the key and value with those from reference features in the late self-attention layers. Our method demonstrates superiority over existing approaches, best reflecting the style of the references and ensuring that resulting images match the text prompts most accurately.
  arXiv  Detail & Related papers  (2024-02-20T12:51:17Z)
• Style Aligned Image Generation via Shared Attention [61.121465570763085]
  We introduce StyleAligned, a technique designed to establish style alignment among a series of generated images. By employing minimal attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. Our method's evaluation across diverse styles and text prompts demonstrates high-quality and fidelity.
  arXiv  Detail & Related papers  (2023-12-04T18:55:35Z)
• Portrait Diffusion: Training-free Face Stylization with Chain-of-Painting [64.43760427752532]
  Face stylization refers to the transformation of a face into a specific portrait style. Current methods require the use of example-based adaptation approaches to fine-tune pre-trained generative models. This paper proposes a training-free face stylization framework, named Portrait Diffusion.
  arXiv  Detail & Related papers  (2023-12-03T06:48:35Z)
• StyleAdapter: A Unified Stylized Image Generation Model [97.24936247688824]
  StyleAdapter is a unified stylized image generation model capable of producing a variety of stylized images. It can be integrated with existing controllable synthesis methods, such as T2I-adapter and ControlNet.
  arXiv  Detail & Related papers  (2023-09-04T19:16:46Z)
• DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization [66.42741426640633]
  DiffStyler is a dual diffusion processing architecture to control the balance between the content and style of diffused results. We propose a content image-based learnable noise on which the reverse denoising process is based, enabling the stylization results to better preserve the structure information of the content image.
  arXiv  Detail & Related papers  (2022-11-19T12:30:44Z)
• Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning [84.8813842101747]
  Contrastive Arbitrary Style Transfer (CAST) is a new style representation learning and style transfer method via contrastive learning. Our framework consists of three key components, i.e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer.
  arXiv  Detail & Related papers  (2022-05-19T13:11:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.