Inversion-Based Style Transfer with Diffusion Models

• URL: http://arxiv.org/abs/2211.13203v3
• Date: Mon, 20 Mar 2023 14:32:01 GMT
• Title: Inversion-Based Style Transfer with Diffusion Models
• Authors: Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, Changsheng Xu
• Abstract summary: Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image.
• Score: 78.93863016223858
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The artistic style within a painting is the means of expression, which includes not only the painting material, colors, and brushstrokes, but also the high-level attributes including semantic elements, object shapes, etc. Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. The pre-trained text-to-image synthesis diffusion probabilistic models have achieved remarkable quality, but it often requires extensive textual descriptions to accurately portray attributes of a particular painting. We believe that the uniqueness of an artwork lies precisely in the fact that it cannot be adequately explained with normal language. Our key idea is to learn artistic style directly from a single painting and then guide the synthesis without providing complex textual descriptions. Specifically, we assume style as a learnable textual description of a painting. We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image, thus capturing and transferring the artistic style of a painting. We demonstrate the quality and efficiency of our method on numerous paintings of various artists and styles. Code and models are available at https://github.com/zyxElsa/InST.

Related papers

• Bridging Text and Image for Artist Style Transfer via Contrastive Learning [21.962361974579036]
  We propose a Contrastive Learning for Artistic Style Transfer (CLAST) to control arbitrary style transfer. We introduce a supervised contrastive training strategy to effectively extract style descriptions from the image-text model. We also propose a novel and efficient adaLN based state space models that explore style-content fusion.
  arXiv  Detail & Related papers  (2024-10-12T15:27:57Z)
• Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt [12.27693060663517]
  Artistic style transfer aims to transfer the learned artistic style onto an arbitrary content image, generating artistic stylized images. We propose a novel pre-trained diffusion-based artistic style transfer method, called LSAST. Our proposed method can generate more highly realistic artistic stylized images than the state-of-the-art artistic style transfer methods.
  arXiv  Detail & Related papers  (2024-04-17T15:28:53Z)
• Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models [47.19481598385283]
  ArtSavant is a tool to determine the unique style of an artist by comparing it to a reference dataset of works from WikiArt. We then perform a large-scale empirical study to provide quantitative insight on the prevalence of artistic style copying across 3 popular text-to-image generative models.
  arXiv  Detail & Related papers  (2024-04-11T17:59:43Z)
• Text-Guided Synthesis of Eulerian Cinemagraphs [81.20353774053768]
  We introduce Text2Cinemagraph, a fully automated method for creating cinemagraphs from text descriptions. We focus on cinemagraphs of fluid elements, such as flowing rivers, and drifting clouds, which exhibit continuous motion and repetitive textures.
  arXiv  Detail & Related papers  (2023-07-06T17:59:31Z)
• Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysis [23.388338598125195]
  GOYA is a method that distills the artistic knowledge captured in a recent generative model to disentangle content and style. Experiments show that synthetically generated images sufficiently serve as a proxy of the real distribution of artworks.
  arXiv  Detail & Related papers  (2023-04-20T13:00:46Z)
• StylerDALLE: Language-Guided Style Transfer Using a Vector-Quantized Tokenizer of a Large-Scale Generative Model [64.26721402514957]
  We propose StylerDALLE, a style transfer method that uses natural language to describe abstract art styles. Specifically, we formulate the language-guided style transfer task as a non-autoregressive token sequence translation. To incorporate style information, we propose a Reinforcement Learning strategy with CLIP-based language supervision.
  arXiv  Detail & Related papers  (2023-03-16T12:44:44Z)
• Language Does More Than Describe: On The Lack Of Figurative Speech in Text-To-Image Models [63.545146807810305]
  Text-to-image diffusion models can generate high-quality pictures from textual input prompts. These models have been trained using text data collected from content-based labelling protocols. We characterise the sentimentality, objectiveness and degree of abstraction of publicly available text data used to train current text-to-image diffusion models.
  arXiv  Detail & Related papers  (2022-10-19T14:20:05Z)
• Interactive Style Transfer: All is Your Palette [74.06681967115594]
  We propose a drawing-like interactive style transfer (IST) method, by which users can interactively create a harmonious-style image. Our IST method can serve as a brush, dip style from anywhere, and then paint to any region of the target content image.
  arXiv  Detail & Related papers  (2022-03-25T06:38:46Z)
• Name Your Style: An Arbitrary Artist-aware Image Style Transfer [38.41608300670523]
  We propose a text-driven image style transfer (TxST) that leverages advanced image-text encoders to control arbitrary style transfer. We introduce a contrastive training strategy to effectively extract style descriptions from the image-text model. We also propose a novel and efficient attention module that explores cross-attentions to fuse style and content features.
  arXiv  Detail & Related papers  (2022-02-28T06:21:38Z)
• What Can Style Transfer and Paintings Do For Model Robustness? [12.543035508615896]
  A common strategy for improving model robustness is through data augmentations. Recent work has shown that arbitrary style transfer can be used as a form of data augmentation. We show that learning from paintings as a form of perceptual data augmentation can improve model robustness.
  arXiv  Detail & Related papers  (2020-11-30T00:25:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.