A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

• URL: http://arxiv.org/abs/2201.08683v1
• Date: Fri, 21 Jan 2022 13:18:16 GMT
• Title: A Comprehensive Study of Vision Transformers on Dense Prediction Tasks
• Authors: Kishaan Jeeveswaran, Senthilkumar Kathiresan, Arnav Varma, Omar Magdy, Bahram Zonooz, and Elahe Arani
• Abstract summary: Convolutional Neural Networks (CNNs) have been the standard choice in vision tasks. Recent studies have shown that Vision Transformers (VTs) achieve comparable performance in challenging tasks such as object detection and semantic segmentation. This poses several questions about their generalizability, robustness, reliability, and texture bias when used to extract features for complex tasks.
• Score: 10.013443811899466
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Convolutional Neural Networks (CNNs), architectures consisting of convolutional layers, have been the standard choice in vision tasks. Recent studies have shown that Vision Transformers (VTs), architectures based on self-attention modules, achieve comparable performance in challenging tasks such as object detection and semantic segmentation. However, the image processing mechanism of VTs is different from that of conventional CNNs. This poses several questions about their generalizability, robustness, reliability, and texture bias when used to extract features for complex tasks. To address these questions, we study and compare VT and CNN architectures as feature extractors in object detection and semantic segmentation. Our extensive empirical results show that the features generated by VTs are more robust to distribution shifts, natural corruptions, and adversarial attacks in both tasks, whereas CNNs perform better at higher image resolutions in object detection. Furthermore, our results demonstrate that VTs in dense prediction tasks produce more reliable and less texture-biased predictions.

Related papers

• Towards Evaluating the Robustness of Visual State Space Models [63.14954591606638]
  Vision State Space Models (VSSMs) have demonstrated remarkable performance in visual perception tasks. However, their robustness under natural and adversarial perturbations remains a critical concern. We present a comprehensive evaluation of VSSMs' robustness under various perturbation scenarios.
  arXiv  Detail & Related papers  (2024-06-13T17:59:44Z)
• A Comparative Survey of Vision Transformers for Feature Extraction in Texture Analysis [9.687982148528187]
  Convolutional Neural Networks (CNNs) are currently among the best texture analysis approaches. Vision Transformers (ViTs) have been surpassing the performance of CNNs on tasks such as object recognition. This work explores various pre-trained ViT architectures when transferred to tasks that rely on textures.
  arXiv  Detail & Related papers  (2024-06-10T09:48:13Z)
• Vision Transformer with Convolutions Architecture Search [72.70461709267497]
  We propose an architecture search method-Vision Transformer with Convolutions Architecture Search (VTCAS) The high-performance backbone network searched by VTCAS introduces the desirable features of convolutional neural networks into the Transformer architecture. It enhances the robustness of the neural network for object recognition, especially in the low illumination indoor scene.
  arXiv  Detail & Related papers  (2022-03-20T02:59:51Z)
• Rich CNN-Transformer Feature Aggregation Networks for Super-Resolution [50.10987776141901]
  Recent vision transformers along with self-attention have achieved promising results on various computer vision tasks. We introduce an effective hybrid architecture for super-resolution (SR) tasks, which leverages local features from CNNs and long-range dependencies captured by transformers. Our proposed method achieves state-of-the-art SR results on numerous benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-15T06:52:25Z)
• Do Vision Transformers See Like Convolutional Neural Networks? [45.69780772718875]
  Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. Are they acting like convolutional networks, or learning entirely different visual representations? We find striking differences between the two architectures, such as ViT having more uniform representations across all layers.
  arXiv  Detail & Related papers  (2021-08-19T17:27:03Z)
• Efficient Training of Visual Transformers with Small-Size Datasets [64.60765211331697]
  Visual Transformers (VTs) are emerging as an architectural paradigm alternative to Convolutional networks (CNNs) We show that, despite having a comparable accuracy when trained on ImageNet, their performance on smaller datasets can be largely different. We propose a self-supervised task which can extract additional information from images with only a negligible computational overhead.
  arXiv  Detail & Related papers  (2021-06-07T16:14:06Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)
• Understanding Robustness of Transformers for Image Classification [34.51672491103555]
  Vision Transformer (ViT) has surpassed ResNets for image classification. Details of the Transformer architecture lead one to wonder whether these networks are as robust. We find that ViT models are at least as robust as the ResNet counterparts on a broad range of perturbations.
  arXiv  Detail & Related papers  (2021-03-26T16:47:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.