ViTGAN: Training GANs with Vision Transformers

• URL: http://arxiv.org/abs/2107.04589v2
• Date: Wed, 29 May 2024 09:41:05 GMT
• Title: ViTGAN: Training GANs with Vision Transformers
• Authors: Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, Ce Liu,
• Abstract summary: Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases. We introduce several novel regularization techniques for training GANs with ViTs. Our approach, named ViTGAN, achieves comparable performance to the leading CNN-based GAN models on three datasets.
• Score: 46.769407314698434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases. In this paper, we investigate if such performance can be extended to image generation. To this end, we integrate the ViT architecture into generative adversarial networks (GANs). For ViT discriminators, we observe that existing regularization methods for GANs interact poorly with self-attention, causing serious instability during training. To resolve this issue, we introduce several novel regularization techniques for training GANs with ViTs. For ViT generators, we examine architectural choices for latent and pixel mapping layers to facilitate convergence. Empirically, our approach, named ViTGAN, achieves comparable performance to the leading CNN-based GAN models on three datasets: CIFAR-10, CelebA, and LSUN bedroom.

Related papers

• When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecture [32.260596998171835]
  Adrial training is still required for ViTs to defend against such adversarial attacks. We find that pre-training and SGD are necessary for ViTs' adversarial training. Our code is available at https://versa.com/mo666666/When-Adrial-Training-Meets-Vision-Transformers.
  arXiv  Detail & Related papers  (2022-10-14T05:37:20Z)
• Self-Distilled Vision Transformer for Domain Generalization [58.76055100157651]
  Vision transformers (ViTs) are challenging the supremacy of CNNs on standard benchmarks. We propose a simple DG approach for ViTs, coined as self-distillation for ViTs. We empirically demonstrate notable performance gains with different DG baselines and various ViT backbones in five challenging datasets.
  arXiv  Detail & Related papers  (2022-07-25T17:57:05Z)
• Deeper Insights into ViTs Robustness towards Common Corruptions [82.79764218627558]
  We investigate how CNN-like architectural designs and CNN-based data augmentation strategies impact on ViTs' robustness towards common corruptions. We demonstrate that overlapping patch embedding and convolutional Feed-Forward Network (FFN) boost performance on robustness. We also introduce a novel conditional method enabling input-varied augmentations from two angles.
  arXiv  Detail & Related papers  (2022-04-26T08:22:34Z)
• Evaluating Vision Transformer Methods for Deep Reinforcement Learning from Pixels [7.426118390008397]
  We evaluate Vision Transformers (ViT) training methods for image-based reinforcement learning control tasks. We compare these results to a leading convolutional-network architecture method, RAD. We find that the CNN architectures trained using RAD still generally provide superior performance.
  arXiv  Detail & Related papers  (2022-04-11T07:10:58Z)
• Improving Vision Transformers by Revisiting High-frequency Components [106.7140968644414]
  We show that Vision Transformer (ViT) models are less effective in capturing the high-frequency components of images than CNN models. To compensate, we propose HAT, which directly augments high-frequency components of images via adversarial training. We show that HAT can consistently boost the performance of various ViT models.
  arXiv  Detail & Related papers  (2022-04-03T05:16:51Z)
• UVCGAN: UNet Vision Transformer cycle-consistent GAN for unpaired image-to-image translation [7.998209482848582]
  Image-to-image translation has broad applications in art, design, and scientific simulations. This work examines if equipping CycleGAN with a vision transformer (ViT) and employing advanced generative adversarial network (GAN) training techniques can achieve better performance.
  arXiv  Detail & Related papers  (2022-03-04T20:27:16Z)
• Hybrid Local-Global Transformer for Image Dehazing [18.468149424220424]
  Vision Transformer (ViT) has shown impressive performance on high-level and low-level vision tasks. We propose a new ViT architecture, named Hybrid Local-Global Vision Transformer (HyLoG-ViT), for single image dehazing.
  arXiv  Detail & Related papers  (2021-09-15T06:13:22Z)
• Long-Short Temporal Contrastive Learning of Video Transformers [62.71874976426988]
  Self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results on par or better than those obtained with supervised pretraining on large-scale image datasets. Our approach, named Long-Short Temporal Contrastive Learning, enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
  arXiv  Detail & Related papers  (2021-06-17T02:30:26Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.