StyleAdapter: A Unified Stylized Image Generation Model

• URL: http://arxiv.org/abs/2309.01770v2
• Date: Wed, 30 Oct 2024 17:05:17 GMT
• Title: StyleAdapter: A Unified Stylized Image Generation Model
• Authors: Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, Ping Luo,
• Abstract summary: StyleAdapter is a unified stylized image generation model capable of producing a variety of stylized images. It can be integrated with existing controllable synthesis methods, such as T2I-adapter and ControlNet.
• Score: 97.24936247688824
• License:
• Abstract: This work focuses on generating high-quality images with specific style of reference images and content of provided textual descriptions. Current leading algorithms, i.e., DreamBooth and LoRA, require fine-tuning for each style, leading to time-consuming and computationally expensive processes. In this work, we propose StyleAdapter, a unified stylized image generation model capable of producing a variety of stylized images that match both the content of a given prompt and the style of reference images, without the need for per-style fine-tuning. It introduces a two-path cross-attention (TPCA) module to separately process style information and textual prompt, which cooperate with a semantic suppressing vision model (SSVM) to suppress the semantic content of style images. In this way, it can ensure that the prompt maintains control over the content of the generated images, while also mitigating the negative impact of semantic information in style references. This results in the content of the generated image adhering to the prompt, and its style aligning with the style references. Besides, our StyleAdapter can be integrated with existing controllable synthesis methods, such as T2I-adapter and ControlNet, to attain a more controllable and stable generation process. Extensive experiments demonstrate the superiority of our method over previous works.

Related papers

• StyleBrush: Style Extraction and Transfer from a Single Image [19.652575295703485]
  Stylization for visual content aims to add specific style patterns at the pixel level while preserving the original structural features. We propose StyleBrush, a method that accurately captures styles from a reference image and brushes'' the extracted style onto other input visual content.
  arXiv  Detail & Related papers  (2024-08-18T14:27:20Z)
• StyleMaster: Towards Flexible Stylized Image Generation with Diffusion Models [42.45078883553856]
  Stylized Text-to-Image Generation (STIG) aims to generate images based on text prompts and style reference images. We in this paper propose a novel framework dubbed as StyleMaster for this task by leveraging pretrained Stable Diffusion. Two objective functions are introduced to optimize the model together with denoising loss, which can further enhance semantic and style consistency.
  arXiv  Detail & Related papers  (2024-05-24T07:19:40Z)
• Visual Style Prompting with Swapping Self-Attention [26.511518230332758]
  We propose a novel approach to produce a diverse range of images while maintaining specific style elements and nuances. During the denoising process, we keep the query from original features while swapping the key and value with those from reference features in the late self-attention layers. Our method demonstrates superiority over existing approaches, best reflecting the style of the references and ensuring that resulting images match the text prompts most accurately.
  arXiv  Detail & Related papers  (2024-02-20T12:51:17Z)
• Style Aligned Image Generation via Shared Attention [61.121465570763085]
  We introduce StyleAligned, a technique designed to establish style alignment among a series of generated images. By employing minimal attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. Our method's evaluation across diverse styles and text prompts demonstrates high-quality and fidelity.
  arXiv  Detail & Related papers  (2023-12-04T18:55:35Z)
• StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter [78.75422651890776]
  StyleCrafter is a generic method that enhances pre-trained T2V models with a style control adapter. To promote content-style disentanglement, we remove style descriptions from the text prompt and extract style information solely from the reference image. StyleCrafter efficiently generates high-quality stylized videos that align with the content of the texts and resemble the style of the reference images.
  arXiv  Detail & Related papers  (2023-12-01T03:53:21Z)
• ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors [105.37795139586075]
  We propose a new task for stylizing'' text-to-image models, namely text-driven stylized image generation. We present a new diffusion model (ControlStyle) via upgrading a pre-trained text-to-image model with a trainable modulation network. Experiments demonstrate the effectiveness of our ControlStyle in producing more visually pleasing and artistic results.
  arXiv  Detail & Related papers  (2023-11-09T15:50:52Z)
• Visual Captioning at Will: Describing Images and Videos Guided by a Few Stylized Sentences [49.66987347397398]
  Few-Shot Stylized Visual Captioning aims to generate captions in any desired style, using only a few examples as guidance during inference. We propose a framework called FS-StyleCap for this task, which utilizes a conditional encoder-decoder language model and a visual projection module.
  arXiv  Detail & Related papers  (2023-07-31T04:26:01Z)
• DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization [66.42741426640633]
  DiffStyler is a dual diffusion processing architecture to control the balance between the content and style of diffused results. We propose a content image-based learnable noise on which the reverse denoising process is based, enabling the stylization results to better preserve the structure information of the content image.
  arXiv  Detail & Related papers  (2022-11-19T12:30:44Z)
• StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation [10.357474047610172]
  We present an approach for generating styled drawings for a given text description where a user can specify a desired drawing style. Inspired by a theory in art that style and content are generally inseparable during the creative process, we propose a coupled approach, known here as StyleCLIPDraw. Based on human evaluation, the styles of images generated by StyleCLIPDraw are strongly preferred to those by the sequential approach.
  arXiv  Detail & Related papers  (2022-02-24T21:03:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.