Sem-CS: Semantic CLIPStyler for Text-Based Image Style Transfer

• URL: http://arxiv.org/abs/2307.05934v1
• Date: Wed, 12 Jul 2023 05:59:42 GMT
• Title: Sem-CS: Semantic CLIPStyler for Text-Based Image Style Transfer
• Authors: Chanda Grover Kamra, Indra Deep Mastan, Debayan Gupta
• Abstract summary: We propose Semantic CLIPStyler (Sem-CS) that performs semantic style transfer. Sem-CS first segments the content image into salient and non-salient objects and then transfers artistic style based on a given style text description. Our empirical results, including DISTS, NIMA and user study scores, show that our proposed framework yields superior qualitative and quantitative performance.
• Score: 4.588028371034406
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: CLIPStyler demonstrated image style transfer with realistic textures using only a style text description (instead of requiring a reference style image). However, the ground semantics of objects in the style transfer output is lost due to style spill-over on salient and background objects (content mismatch) or over-stylization. To solve this, we propose Semantic CLIPStyler (Sem-CS), that performs semantic style transfer. Sem-CS first segments the content image into salient and non-salient objects and then transfers artistic style based on a given style text description. The semantic style transfer is achieved using global foreground loss (for salient objects) and global background loss (for non-salient objects). Our empirical results, including DISTS, NIMA and user study scores, show that our proposed framework yields superior qualitative and quantitative performance. Our code is available at github.com/chandagrover/sem-cs.

Related papers

• FAGStyle: Feature Augmentation on Geodesic Surface for Zero-shot Text-guided Diffusion Image Style Transfer [2.3293561091456283]
  The goal of image style transfer is to render an image guided by a style reference while maintaining the original content. We introduce FAGStyle, a zero-shot text-guided diffusion image style transfer method. Our approach enhances inter-patch information interaction by incorporating the Sliding Window Crop technique.
  arXiv  Detail & Related papers  (2024-08-20T04:20:11Z)
• Soulstyler: Using Large Language Model to Guide Image Style Transfer for Target Object [9.759321877363258]
  "Soulstyler" allows users to guide the stylization of specific objects in an image through simple textual descriptions. We introduce a large language model to parse the text and identify stylization goals and specific styles. We also introduce a novel localized text-image block matching loss that ensures that style transfer is performed only on specified target objects.
  arXiv  Detail & Related papers  (2023-11-22T18:15:43Z)
• StylerDALLE: Language-Guided Style Transfer Using a Vector-Quantized Tokenizer of a Large-Scale Generative Model [64.26721402514957]
  We propose StylerDALLE, a style transfer method that uses natural language to describe abstract art styles. Specifically, we formulate the language-guided style transfer task as a non-autoregressive token sequence translation. To incorporate style information, we propose a Reinforcement Learning strategy with CLIP-based language supervision.
  arXiv  Detail & Related papers  (2023-03-16T12:44:44Z)
• SEM-CS: Semantic CLIPStyler for Text-Based Image Style Transfer [4.588028371034406]
  We propose Semantic CLIPStyler (Sem-CS) that performs semantic style transfer. Sem-CS first segments the content image into salient and non-salient objects and then transfers artistic style based on a given style text description. Our empirical results, including DISTS, NIMA and user study scores, show that our proposed framework yields superior qualitative and quantitative performance.
  arXiv  Detail & Related papers  (2023-03-11T07:33:06Z)
• DSI2I: Dense Style for Unpaired Image-to-Image Translation [70.93865212275412]
  Unpaired exemplar-based image-to-image (UEI2I) translation aims to translate a source image to a target image domain with the style of a target image exemplar. We propose to represent style as a dense feature map, allowing for a finer-grained transfer to the source image without requiring any external semantic information. Our results show that the translations produced by our approach are more diverse, preserve the source content better, and are closer to the exemplars when compared to the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-12-26T18:45:25Z)
• Diffusion-based Image Translation using Disentangled Style and Content Representation [51.188396199083336]
  Diffusion-based image translation guided by semantic texts or a single target image has enabled flexible style transfer. It is often difficult to maintain the original content of the image during the reverse diffusion. We present a novel diffusion-based unsupervised image translation method using disentangled style and content representation. Our experimental results show that the proposed method outperforms state-of-the-art baseline models in both text-guided and image-guided translation tasks.
  arXiv  Detail & Related papers  (2022-09-30T06:44:37Z)
• Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning [84.8813842101747]
  Contrastive Arbitrary Style Transfer (CAST) is a new style representation learning and style transfer method via contrastive learning. Our framework consists of three key components, i.e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer.
  arXiv  Detail & Related papers  (2022-05-19T13:11:24Z)
• CLIPstyler: Image Style Transfer with a Single Text Condition [34.24876359759408]
  Existing neural style transfer methods require reference style images to transfer texture information of style images to content images. We propose a new framework that enables a style transfer without' a style image, but only with a text description of the desired style.
  arXiv  Detail & Related papers  (2021-12-01T09:48:53Z)
• Context-Aware Image Inpainting with Learned Semantic Priors [100.99543516733341]
  We introduce pretext tasks that are semantically meaningful to estimating the missing contents. We propose a context-aware image inpainting model, which adaptively integrates global semantics and local features.
  arXiv  Detail & Related papers  (2021-06-14T08:09:43Z)
• Language-Driven Image Style Transfer [72.36790598245096]
  We introduce a new task -- language-driven image style transfer (textttLDIST) -- to manipulate the style of a content image, guided by a text. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. Experiments show that our CLVA is effective and achieves superb transferred results on textttLDIST.
  arXiv  Detail & Related papers  (2021-06-01T01:58:50Z)
• DeepObjStyle: Deep Object-based Photo Style Transfer [31.75300124593133]
  One of the major challenges of style transfer is the appropriate image features supervision between the output image and the input (style and content) images. We propose an object-based style transfer approach, called DeepStyle, for the style supervision in the training data-independent framework.
  arXiv  Detail & Related papers  (2020-12-11T17:02:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.