StyleDrop: Text-to-Image Generation in Any Style

• URL: http://arxiv.org/abs/2306.00983v1
• Date: Thu, 1 Jun 2023 17:59:51 GMT
• Title: StyleDrop: Text-to-Image Generation in Any Style
• Authors: Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, Yuan Hao, Irfan Essa, Michael Rubinstein, Dilip Krishnan
• Abstract summary: StyleDrop is a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. It efficiently learns a new style by fine-tuning very few trainable parameters and improving the quality via iterative training. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse convincingly outperforms other methods.
• Score: 43.42391701778596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained large text-to-image models synthesize impressive images with an appropriate use of text prompts. However, ambiguities inherent in natural language and out-of-distribution effects make it hard to synthesize image styles, that leverage a specific design pattern, texture or material. In this paper, we introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. The proposed method is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than $1\%$ of total model parameters) and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. More results are available at our project website: https://styledrop.github.io

Related papers

• Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics [3.9717825324709413]
  Style has been primarily considered in terms of artistic elements such as colors, brushstrokes, and lighting. In this study, we propose a zero-shot scheme for image variation with coordinated semantics.
  arXiv  Detail & Related papers  (2024-10-24T08:34:57Z)
• StyleTokenizer: Defining Image Style by a Single Instance for Controlling Diffusion Models [38.31347002106355]
  StyleTokenizer is a zero-shot style control image generation method. It aligns style representation with text representation using a style tokenizer. This alignment effectively minimizes the impact on the effectiveness of text prompts.
  arXiv  Detail & Related papers  (2024-09-04T09:01:21Z)
• StyleShot: A Snapshot on Any Style [20.41380860802149]
  We show that, a good style representation is crucial and sufficient for generalized style transfer without test-time tuning. We achieve this through constructing a style-aware encoder and a well-organized style dataset called StyleGallery. We highlight that, our approach, named StyleShot, is simple yet effective in mimicking various desired styles, without test-time tuning.
  arXiv  Detail & Related papers  (2024-07-01T16:05:18Z)
• Style Aligned Image Generation via Shared Attention [61.121465570763085]
  We introduce StyleAligned, a technique designed to establish style alignment among a series of generated images. By employing minimal attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. Our method's evaluation across diverse styles and text prompts demonstrates high-quality and fidelity.
  arXiv  Detail & Related papers  (2023-12-04T18:55:35Z)
• Visual Captioning at Will: Describing Images and Videos Guided by a Few Stylized Sentences [49.66987347397398]
  Few-Shot Stylized Visual Captioning aims to generate captions in any desired style, using only a few examples as guidance during inference. We propose a framework called FS-StyleCap for this task, which utilizes a conditional encoder-decoder language model and a visual projection module.
  arXiv  Detail & Related papers  (2023-07-31T04:26:01Z)
• Deep Image Style Transfer from Freeform Text [4.186575888568896]
  This paper creates a novel method of deep neural style transfer by generating style images from freeform user text input. The language model and style transfer model form a seamless pipeline that can create output images with similar losses and improved quality.
  arXiv  Detail & Related papers  (2022-12-13T19:24:08Z)
• DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization [66.42741426640633]
  DiffStyler is a dual diffusion processing architecture to control the balance between the content and style of diffused results. We propose a content image-based learnable noise on which the reverse denoising process is based, enabling the stylization results to better preserve the structure information of the content image.
  arXiv  Detail & Related papers  (2022-11-19T12:30:44Z)
• Arbitrary Style Guidance for Enhanced Diffusion-Based Text-to-Image Generation [13.894251782142584]
  Diffusion-based text-to-image generation models like GLIDE and DALLE-2 have gained wide success recently. We propose a novel style guidance method to support generating images using arbitrary style guided by a reference image.
  arXiv  Detail & Related papers  (2022-11-14T20:52:57Z)
• Learning Diverse Tone Styles for Image Retouching [73.60013618215328]
  We propose to learn diverse image retouching with normalizing flow-based architectures. A joint-training pipeline is composed of a style encoder, a conditional RetouchNet, and the image tone style normalizing flow (TSFlow) module. Our proposed method performs favorably against state-of-the-art methods and is effective in generating diverse results.
  arXiv  Detail & Related papers  (2022-07-12T09:49:21Z)
• Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning [84.8813842101747]
  Contrastive Arbitrary Style Transfer (CAST) is a new style representation learning and style transfer method via contrastive learning. Our framework consists of three key components, i.e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer.
  arXiv  Detail & Related papers  (2022-05-19T13:11:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.