Bridging Text and Image for Artist Style Transfer via Contrastive Learning

• URL: http://arxiv.org/abs/2410.09566v1
• Date: Sat, 12 Oct 2024 15:27:57 GMT
• Title: Bridging Text and Image for Artist Style Transfer via Contrastive Learning
• Authors: Zhi-Song Liu, Li-Wen Wang, Jun Xiao, Vicky Kalogeiton,
• Abstract summary: We propose a Contrastive Learning for Artistic Style Transfer (CLAST) to control arbitrary style transfer. We introduce a supervised contrastive training strategy to effectively extract style descriptions from the image-text model. We also propose a novel and efficient adaLN based state space models that explore style-content fusion.
• Score: 21.962361974579036
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image style transfer has attracted widespread attention in the past few years. Despite its remarkable results, it requires additional style images available as references, making it less flexible and inconvenient. Using text is the most natural way to describe the style. More importantly, text can describe implicit abstract styles, like styles of specific artists or art movements. In this paper, we propose a Contrastive Learning for Artistic Style Transfer (CLAST) that leverages advanced image-text encoders to control arbitrary style transfer. We introduce a supervised contrastive training strategy to effectively extract style descriptions from the image-text model (i.e., CLIP), which aligns stylization with the text description. To this end, we also propose a novel and efficient adaLN based state space models that explore style-content fusion. Finally, we achieve a text-driven image style transfer. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods in artistic style transfer. More importantly, it does not require online fine-tuning and can render a 512x512 image in 0.03s.

Related papers

• StyleMamba : State Space Model for Efficient Text-driven Image Style Transfer [9.010012117838725]
  StyleMamba is an efficient image style transfer framework that translates text prompts into corresponding visual styles. Existing text-guided stylization requires hundreds of training iterations and takes a lot of computing resources.
  arXiv  Detail & Related papers  (2024-05-08T12:57:53Z)
• Style Aligned Image Generation via Shared Attention [61.121465570763085]
  We introduce StyleAligned, a technique designed to establish style alignment among a series of generated images. By employing minimal attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. Our method's evaluation across diverse styles and text prompts demonstrates high-quality and fidelity.
  arXiv  Detail & Related papers  (2023-12-04T18:55:35Z)
• Any-to-Any Style Transfer: Making Picasso and Da Vinci Collaborate [58.83278629019384]
  Style transfer aims to render the style of a given image for style reference to another given image for content reference. Existing approaches either apply the holistic style of the style image in a global manner, or migrate local colors and textures of the style image to the content counterparts in a pre-defined way. We propose Any-to-Any Style Transfer, which enables users to interactively select styles of regions in the style image and apply them to the prescribed content regions.
  arXiv  Detail & Related papers  (2023-04-19T15:15:36Z)
• StylerDALLE: Language-Guided Style Transfer Using a Vector-Quantized Tokenizer of a Large-Scale Generative Model [64.26721402514957]
  We propose StylerDALLE, a style transfer method that uses natural language to describe abstract art styles. Specifically, we formulate the language-guided style transfer task as a non-autoregressive token sequence translation. To incorporate style information, we propose a Reinforcement Learning strategy with CLIP-based language supervision.
  arXiv  Detail & Related papers  (2023-03-16T12:44:44Z)
• ITstyler: Image-optimized Text-based Style Transfer [25.60521982742093]
  We present a text-based style transfer method that does not require optimization at the inference stage. Specifically, we convert text input to the style space of the pre-trained VGG network to realize a more effective style swap. Our method can transfer arbitrary new styles of text input in real-time and synthesize high-quality artistic images.
  arXiv  Detail & Related papers  (2023-01-26T03:08:43Z)
• Inversion-Based Style Transfer with Diffusion Models [78.93863016223858]
  Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image.
  arXiv  Detail & Related papers  (2022-11-23T18:44:25Z)
• DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization [66.42741426640633]
  DiffStyler is a dual diffusion processing architecture to control the balance between the content and style of diffused results. We propose a content image-based learnable noise on which the reverse denoising process is based, enabling the stylization results to better preserve the structure information of the content image.
  arXiv  Detail & Related papers  (2022-11-19T12:30:44Z)
• Name Your Style: An Arbitrary Artist-aware Image Style Transfer [38.41608300670523]
  We propose a text-driven image style transfer (TxST) that leverages advanced image-text encoders to control arbitrary style transfer. We introduce a contrastive training strategy to effectively extract style descriptions from the image-text model. We also propose a novel and efficient attention module that explores cross-attentions to fuse style and content features.
  arXiv  Detail & Related papers  (2022-02-28T06:21:38Z)
• CLIPstyler: Image Style Transfer with a Single Text Condition [34.24876359759408]
  Existing neural style transfer methods require reference style images to transfer texture information of style images to content images. We propose a new framework that enables a style transfer without' a style image, but only with a text description of the desired style.
  arXiv  Detail & Related papers  (2021-12-01T09:48:53Z)
• Language-Driven Image Style Transfer [72.36790598245096]
  We introduce a new task -- language-driven image style transfer (textttLDIST) -- to manipulate the style of a content image, guided by a text. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. Experiments show that our CLVA is effective and achieves superb transferred results on textttLDIST.
  arXiv  Detail & Related papers  (2021-06-01T01:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.