Fine-Grained Image Style Transfer with Visual Transformers

• URL: http://arxiv.org/abs/2210.05176v1
• Date: Tue, 11 Oct 2022 06:26:00 GMT
• Title: Fine-Grained Image Style Transfer with Visual Transformers
• Authors: Jianbo Wang, Huan Yang, Jianlong Fu, Toshihiko Yamasaki, and Baining Guo
• Abstract summary: We propose a novel STyle TRansformer (STTR) network which breaks both content and style images into visual tokens to achieve a fine-grained style transformation. To compare STTR with existing approaches, we conduct user studies on Amazon Mechanical Turk.
• Score: 59.85619519384446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the development of the convolutional neural network, image style transfer has drawn increasing attention. However, most existing approaches adopt a global feature transformation to transfer style patterns into content images (e.g., AdaIN and WCT). Such a design usually destroys the spatial information of the input images and fails to transfer fine-grained style patterns into style transfer results. To solve this problem, we propose a novel STyle TRansformer (STTR) network which breaks both content and style images into visual tokens to achieve a fine-grained style transformation. Specifically, two attention mechanisms are adopted in our STTR. We first propose to use self-attention to encode content and style tokens such that similar tokens can be grouped and learned together. We then adopt cross-attention between content and style tokens that encourages fine-grained style transformations. To compare STTR with existing approaches, we conduct user studies on Amazon Mechanical Turk (AMT), which are carried out with 50 human subjects with 1,000 votes in total. Extensive evaluations demonstrate the effectiveness and efficiency of the proposed STTR in generating visually pleasing style transfer results.

Related papers

• Patch Is Not All You Need [57.290256181083016]
  We propose a novel Pattern Transformer to adaptively convert images to pattern sequences for Transformer input. We employ the Convolutional Neural Network to extract various patterns from the input image. We have accomplished state-of-the-art performance on CIFAR-10 and CIFAR-100, and have achieved competitive results on ImageNet.
  arXiv  Detail & Related papers  (2023-08-21T13:54:00Z)
• Improving the Transferability of Adversarial Examples with Arbitrary Style Transfer [32.644062141738246]
  A style transfer network can alter the distribution of low-level visual features in an image while preserving semantic content for humans. We propose a novel attack method named Style Transfer Method (STM) that utilizes a proposed arbitrary style transfer network to transform the images into different domains. Our proposed method can significantly improve the adversarial transferability on either normally trained models or adversarially trained models.
  arXiv  Detail & Related papers  (2023-08-21T09:58:13Z)
• A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning [84.8813842101747]
  Unified Contrastive Arbitrary Style Transfer (UCAST) is a novel style representation learning and transfer framework. We present an adaptive contrastive learning scheme for style transfer by introducing an input-dependent temperature. Our framework consists of three key components, i.e., a parallel contrastive learning scheme for style representation and style transfer, a domain enhancement module for effective learning of style distribution, and a generative network for style transfer.
  arXiv  Detail & Related papers  (2023-03-09T04:35:00Z)
• Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning [84.8813842101747]
  Contrastive Arbitrary Style Transfer (CAST) is a new style representation learning and style transfer method via contrastive learning. Our framework consists of three key components, i.e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer.
  arXiv  Detail & Related papers  (2022-05-19T13:11:24Z)
• Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers [51.581926074686535]
  We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. The proposed TokenGAN has achieved state-of-the-art results on several widely-used image synthesis benchmarks.
  arXiv  Detail & Related papers  (2021-11-05T12:57:50Z)
• StyTr^2: Unbiased Image Style Transfer with Transformers [59.34108877969477]
  The goal of image style transfer is to render an image with artistic features guided by a style reference while maintaining the original content. Traditional neural style transfer methods are usually biased and content leak can be observed by running several times of the style transfer process with the same reference image. We propose a transformer-based approach, namely StyTr2, to address this critical issue.
  arXiv  Detail & Related papers  (2021-05-30T15:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.