Generative Artisan: A Semantic-Aware and Controllable CLIPstyler

• URL: http://arxiv.org/abs/2207.11598v1
• Date: Sat, 23 Jul 2022 20:26:47 GMT
• Title: Generative Artisan: A Semantic-Aware and Controllable CLIPstyler
• Authors: Zhenling Yang, Huacheng Song, Qiunan Wu
• Abstract summary: We present a novel framework that uses a pre-trained CLIP text-image embedding model and guides image style transfer through an FCN semantic segmentation network. Specifically, we solve the portrait over-styling problem for both selfies and real-world landscape with human subjects photos.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recall that most of the current image style transfer methods require the user to give an image of a particular style and then extract that styling feature and texture to generate the style of an image, but there are still some problems: the user may not have a reference style image, or it may be difficult to summarise the desired style in mind with just one image. The recently proposed CLIPstyler has solved this problem, which is able to perform style transfer based only on the provided description of the style image. Although CLIPstyler can achieve good performance when landscapes or portraits appear alone, it can blur the people and lose the original semantics when people and landscapes coexist. Based on these issues, we demonstrate a novel framework that uses a pre-trained CLIP text-image embedding model and guides image style transfer through an FCN semantic segmentation network. Specifically, we solve the portrait over-styling problem for both selfies and real-world landscape with human subjects photos, enhance the contrast between the effect of style transfer in portrait and landscape, and make the degree of image style transfer in different semantic parts fully controllable. Our Generative Artisan resolve the failure case of CLIPstyler and yield both qualitative and quantitative methods to prove ours have much better results than CLIPstyler in both selfies and real-world landscape with human subjects photos. This improvement makes it possible to commercialize our framework for business scenarios such as retouching graphics software.

Related papers

• PS-StyleGAN: Illustrative Portrait Sketching using Attention-Based Style Adaptation [0.0]
  Portrait sketching involves capturing identity specific attributes of a real face with abstract lines and shades. This paper introduces textbfPortrait Sketching StyleGAN (PS-StyleGAN), a style transfer approach tailored for portrait sketch synthesis. We leverage the semantic $W+$ latent space of StyleGAN to generate portrait sketches, allowing us to make meaningful edits, like pose and expression alterations, without compromising identity.
  arXiv  Detail & Related papers  (2024-08-31T04:22:45Z)
• MOSAIC: Multi-Object Segmented Arbitrary Stylization Using CLIP [0.0]
  Style transfer driven by text prompts paved a new path for creatively stylizing the images without collecting an actual style image. We propose a new method Multi-Object Segmented Arbitrary Stylization Using CLIP (MOSAIC) that can apply styles to different objects in the image based on the context extracted from the input prompt. Our method can extend to any arbitrary objects, styles and produce high-quality images compared to the current state of art methods.
  arXiv  Detail & Related papers  (2023-09-24T18:24:55Z)
• StyleAdapter: A Unified Stylized Image Generation Model [97.24936247688824]
  StyleAdapter is a unified stylized image generation model capable of producing a variety of stylized images. It can be integrated with existing controllable synthesis methods, such as T2I-adapter and ControlNet.
  arXiv  Detail & Related papers  (2023-09-04T19:16:46Z)
• Any-to-Any Style Transfer: Making Picasso and Da Vinci Collaborate [58.83278629019384]
  Style transfer aims to render the style of a given image for style reference to another given image for content reference. Existing approaches either apply the holistic style of the style image in a global manner, or migrate local colors and textures of the style image to the content counterparts in a pre-defined way. We propose Any-to-Any Style Transfer, which enables users to interactively select styles of regions in the style image and apply them to the prescribed content regions.
  arXiv  Detail & Related papers  (2023-04-19T15:15:36Z)
• Learning Diverse Tone Styles for Image Retouching [73.60013618215328]
  We propose to learn diverse image retouching with normalizing flow-based architectures. A joint-training pipeline is composed of a style encoder, a conditional RetouchNet, and the image tone style normalizing flow (TSFlow) module. Our proposed method performs favorably against state-of-the-art methods and is effective in generating diverse results.
  arXiv  Detail & Related papers  (2022-07-12T09:49:21Z)
• CCPL: Contrastive Coherence Preserving Loss for Versatile Style Transfer [58.020470877242865]
  We devise a universally versatile style transfer method capable of performing artistic, photo-realistic, and video style transfer jointly. We make a mild and reasonable assumption that global inconsistency is dominated by local inconsistencies and devise a generic Contrastive Coherence Preserving Loss (CCPL) applied to local patches. CCPL can preserve the coherence of the content source during style transfer without degrading stylization.
  arXiv  Detail & Related papers  (2022-07-11T12:09:41Z)
• Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning [84.8813842101747]
  Contrastive Arbitrary Style Transfer (CAST) is a new style representation learning and style transfer method via contrastive learning. Our framework consists of three key components, i.e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer.
  arXiv  Detail & Related papers  (2022-05-19T13:11:24Z)
• StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation [10.357474047610172]
  We present an approach for generating styled drawings for a given text description where a user can specify a desired drawing style. Inspired by a theory in art that style and content are generally inseparable during the creative process, we propose a coupled approach, known here as StyleCLIPDraw. Based on human evaluation, the styles of images generated by StyleCLIPDraw are strongly preferred to those by the sequential approach.
  arXiv  Detail & Related papers  (2022-02-24T21:03:51Z)
• CLIPstyler: Image Style Transfer with a Single Text Condition [34.24876359759408]
  Existing neural style transfer methods require reference style images to transfer texture information of style images to content images. We propose a new framework that enables a style transfer without' a style image, but only with a text description of the desired style.
  arXiv  Detail & Related papers  (2021-12-01T09:48:53Z)
• Language-Driven Image Style Transfer [72.36790598245096]
  We introduce a new task -- language-driven image style transfer (textttLDIST) -- to manipulate the style of a content image, guided by a text. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. Experiments show that our CLVA is effective and achieves superb transferred results on textttLDIST.
  arXiv  Detail & Related papers  (2021-06-01T01:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.