AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer

• URL: http://arxiv.org/abs/2108.03647v2
• Date: Wed, 11 Aug 2021 13:14:49 GMT
• Title: AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer
• Authors: Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, Errui Ding
• Abstract summary: We propose a novel attention and normalization module, named Adaptive Attention Normalization (AdaAttN) Specifically, spatial attention score is learnt from both shallow and deep features of content and style images. Per-point weighted statistics are calculated by regarding a style feature point as a distribution of attention-weighted output of all style feature points. Finally, the content feature is normalized so that they demonstrate the same local feature statistics as the calculated per-point weighted style feature statistics.
• Score: 44.08659730413871
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fast arbitrary neural style transfer has attracted widespread attention from academic, industrial and art communities due to its flexibility in enabling various applications. Existing solutions either attentively fuse deep style feature into deep content feature without considering feature distributions, or adaptively normalize deep content feature according to the style such that their global statistics are matched. Although effective, leaving shallow feature unexplored and without locally considering feature statistics, they are prone to unnatural output with unpleasing local distortions. To alleviate this problem, in this paper, we propose a novel attention and normalization module, named Adaptive Attention Normalization (AdaAttN), to adaptively perform attentive normalization on per-point basis. Specifically, spatial attention score is learnt from both shallow and deep features of content and style images. Then per-point weighted statistics are calculated by regarding a style feature point as a distribution of attention-weighted output of all style feature points. Finally, the content feature is normalized so that they demonstrate the same local feature statistics as the calculated per-point weighted style feature statistics. Besides, a novel local feature loss is derived based on AdaAttN to enhance local visual quality. We also extend AdaAttN to be ready for video style transfer with slight modifications. Experiments demonstrate that our method achieves state-of-the-art arbitrary image/video style transfer. Codes and models are available.

Related papers

• The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach [11.524573224123905]
  CNNs make decisions based on narrow, specific regions of input images. This behavior can severely compromise the model's generalization capabilities. We introduce Saliency Guided Dropout (SGDrop) to address this specific issue.
  arXiv  Detail & Related papers  (2024-09-25T21:30:16Z)
• Unleash the Power of Local Representations for Few-Shot Classification [6.722306005855269]
  Generalizing to novel classes unseen during training is a key challenge of few-shot classification. Recent metric-based methods try to address this by local representations. In this work, we unleash the power of local representations in improving novel-class generalization.
  arXiv  Detail & Related papers  (2024-07-02T05:51:04Z)
• Condition-Invariant Semantic Segmentation [77.10045325743644]
  We implement Condition-Invariant Semantic (CISS) on the current state-of-the-art domain adaptation architecture. Our method achieves the second-best performance on the normal-to-adverse Cityscapes$to$ACDC benchmark. CISS is shown to generalize well to domains unseen during training, such as BDD100K-night and ACDC-night.
  arXiv  Detail & Related papers  (2023-05-27T03:05:07Z)
• Randomized Adversarial Style Perturbations for Domain Generalization [49.888364462991234]
  We propose a novel domain generalization technique, referred to as Randomized Adversarial Style Perturbation (RASP) The proposed algorithm perturbs the style of a feature in an adversarial direction towards a randomly selected class, and makes the model learn against being misled by the unexpected styles observed in unseen target domains. We evaluate the proposed algorithm via extensive experiments on various benchmarks and show that our approach improves domain generalization performance, especially in large-scale benchmarks.
  arXiv  Detail & Related papers  (2023-04-04T17:07:06Z)
• All-to-key Attention for Arbitrary Style Transfer [98.83954812536521]
  We propose a novel all-to-key attention mechanism -- each position of content features is matched to stable key positions of style features. The resultant module, dubbed StyA2K, shows extraordinary performance in preserving the semantic structure and rendering consistent style patterns.
  arXiv  Detail & Related papers  (2022-12-08T06:46:35Z)
• Learning Graph Neural Networks for Image Style Transfer [131.73237185888215]
  State-of-the-art parametric and non-parametric style transfer approaches are prone to either distorted local style patterns due to global statistics alignment, or unpleasing artifacts resulting from patch mismatching. In this paper, we study a novel semi-parametric neural style transfer framework that alleviates the deficiency of both parametric and non-parametric stylization.
  arXiv  Detail & Related papers  (2022-07-24T07:41:31Z)
• SAFIN: Arbitrary Style Transfer With Self-Attentive Factorized Instance Normalization [71.85169368997738]
  Artistic style transfer aims to transfer the style characteristics of one image onto another image while retaining its content. Self-Attention-based approaches have tackled this issue with partial success but suffer from unwanted artifacts. This paper aims to combine the best of both worlds: self-attention and normalization.
  arXiv  Detail & Related papers  (2021-05-13T08:01:01Z)
• In the light of feature distributions: moment matching for Neural Style Transfer [27.25600860698314]
  Style transfer aims to render the content of a given image in the graphical/artistic style of another image. We show that most current implementations of that concept have important theoretical and practical limitations. We propose a novel approach that matches the desired style more precisely, while still being computationally efficient.
  arXiv  Detail & Related papers  (2021-03-12T11:00:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.