GenFormer -- Generated Images are All You Need to Improve Robustness of Transformers on Small Datasets

• URL: http://arxiv.org/abs/2408.14131v2
• Date: Tue, 27 Aug 2024 05:54:42 GMT
• Title: GenFormer -- Generated Images are All You Need to Improve Robustness of Transformers on Small Datasets
• Authors: Sven Oehri, Nikolas Ebert, Ahmed Abdullah, Didier Stricker, Oliver Wasenmüller,
• Abstract summary: We propose GenFormer, a data augmentation strategy utilizing generated images to improve transformer accuracy and robustness on small-scale image classification tasks. In our comprehensive evaluation we propose Tiny ImageNetV2, -R, and -A as new test set variants of Tiny ImageNet. We prove the effectiveness of our approach under challenging conditions with limited training data, demonstrating significant improvements in both accuracy and robustness.
• Score: 11.343905946690352
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent studies showcase the competitive accuracy of Vision Transformers (ViTs) in relation to Convolutional Neural Networks (CNNs), along with their remarkable robustness. However, ViTs demand a large amount of data to achieve adequate performance, which makes their application to small datasets challenging, falling behind CNNs. To overcome this, we propose GenFormer, a data augmentation strategy utilizing generated images, thereby improving transformer accuracy and robustness on small-scale image classification tasks. In our comprehensive evaluation we propose Tiny ImageNetV2, -R, and -A as new test set variants of Tiny ImageNet by transferring established ImageNet generalization and robustness benchmarks to the small-scale data domain. Similarly, we introduce MedMNIST-C and EuroSAT-C as corrupted test set variants of established fine-grained datasets in the medical and aerial domain. Through a series of experiments conducted on small datasets of various domains, including Tiny ImageNet, CIFAR, EuroSAT and MedMNIST datasets, we demonstrate the synergistic power of our method, in particular when combined with common train and test time augmentations, knowledge distillation, and architectural design choices. Additionally, we prove the effectiveness of our approach under challenging conditions with limited training data, demonstrating significant improvements in both accuracy and robustness, bridging the gap between CNNs and ViTs in the small-scale dataset domain.

Related papers

• Optimizing Vision Transformers with Data-Free Knowledge Transfer [8.323741354066474]
  Vision transformers (ViTs) have excelled in various computer vision tasks due to their superior ability to capture long-distance dependencies. We propose compressing large ViT models using Knowledge Distillation (KD), which is implemented data-free to circumvent limitations related to data availability.
  arXiv  Detail & Related papers  (2024-08-12T07:03:35Z)
• Masked autoencoders are effective solution to transformer data-hungry [0.0]
  Vision Transformers (ViTs) outperforms convolutional neural networks (CNNs) in several vision tasks with its global modeling capabilities. ViT lacks the inductive bias inherent to convolution making it require a large amount of data for training. Masked autoencoders (MAE) can make the transformer focus more on the image itself.
  arXiv  Detail & Related papers  (2022-12-12T03:15:19Z)
• How to Train Vision Transformer on Small-scale Datasets? [4.56717163175988]
  In contrast to convolutional neural networks, Vision Transformer lacks inherent inductive biases. We show that self-supervised inductive biases can be learned directly from small-scale datasets. This allows to train these models without large-scale pre-training, changes to model architecture or loss functions.
  arXiv  Detail & Related papers  (2022-10-13T17:59:19Z)
• Impact of Scaled Image on Robustness of Deep Neural Networks [0.0]
  Scaling the raw images creates out-of-distribution data, which makes it a possible adversarial attack to fool the networks. In this work, we propose a Scaling-distortion dataset ImageNet-CS by Scaling a subset of the ImageNet Challenge dataset by different multiples.
  arXiv  Detail & Related papers  (2022-09-02T08:06:58Z)
• ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond [76.35955924137986]
  We propose a Vision Transformer Advanced by Exploring intrinsic IB from convolutions, i.e., ViTAE. ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context. We obtain the state-of-the-art classification performance, i.e., 88.5% Top-1 classification accuracy on ImageNet validation set and the best 91.2% Top-1 accuracy on ImageNet real validation set.
  arXiv  Detail & Related papers  (2022-02-21T10:40:05Z)
• Training Vision Transformers with Only 2040 Images [35.86457465241119]
  Vision Transformers (ViTs) is emerging as an alternative to convolutional neural networks (CNNs) for visual recognition. We give theoretical analyses that our method is superior to other methods in that it can capture both feature alignment and instance similarities. We achieve state-of-the-art results when training from scratch on 7 small datasets under various ViT backbones.
  arXiv  Detail & Related papers  (2022-01-26T03:22:08Z)
• Feature transforms for image data augmentation [74.12025519234153]
  In image classification, many augmentation approaches utilize simple image manipulation algorithms. In this work, we build ensembles on the data level by adding images generated by combining fourteen augmentation approaches. Pretrained ResNet50 networks are finetuned on training sets that include images derived from each augmentation method.
  arXiv  Detail & Related papers  (2022-01-24T14:12:29Z)
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers [74.06040005144382]
  Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. We train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset.
  arXiv  Detail & Related papers  (2021-06-18T17:58:20Z)
• Efficient Training of Visual Transformers with Small-Size Datasets [64.60765211331697]
  Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs) We show that, despite having a comparable accuracy when trained on ImageNet, their performance on smaller datasets can be largely different. We propose a self-supervised task which can extract additional information from images with only a negligible computational overhead.
  arXiv  Detail & Related papers  (2021-06-07T16:14:06Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)
• On Robustness and Transferability of Convolutional Neural Networks [147.71743081671508]
  Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts. We study the interplay between out-of-distribution and transfer performance of modern image classification CNNs for the first time. We find that increasing both the training set and model sizes significantly improve the distributional shift robustness.
  arXiv  Detail & Related papers  (2020-07-16T18:39:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.