HSViT: Horizontally Scalable Vision Transformer

• URL: http://arxiv.org/abs/2404.05196v2
• Date: Tue, 16 Jul 2024 01:04:36 GMT
• Title: HSViT: Horizontally Scalable Vision Transformer
• Authors: Chenhao Xu, Chang-Tsun Li, Chee Peng Lim, Douglas Creighton,
• Abstract summary: Vision Transformer (ViT) requires pre-training on large-scale datasets to perform well. This paper introduces a novel horizontally scalable vision transformer (HSViT) scheme. HSViT achieves up to 10% higher top-1 accuracy than state-of-the-art schemes on small datasets.
• Score: 16.46308352393693
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Due to its deficiency in prior knowledge (inductive bias), Vision Transformer (ViT) requires pre-training on large-scale datasets to perform well. Moreover, the growing layers and parameters in ViT models impede their applicability to devices with limited computing resources. To mitigate the aforementioned challenges, this paper introduces a novel horizontally scalable vision transformer (HSViT) scheme. Specifically, a novel image-level feature embedding is introduced to ViT, where the preserved inductive bias allows the model to eliminate the need for pre-training while outperforming on small datasets. Besides, a novel horizontally scalable architecture is designed, facilitating collaborative model training and inference across multiple computing devices. The experimental results depict that, without pre-training, HSViT achieves up to 10% higher top-1 accuracy than state-of-the-art schemes on small datasets, while providing existing CNN backbones up to 3.1% improvement in top-1 accuracy on ImageNet. The code is available at https://github.com/xuchenhao001/HSViT.

Related papers

• HTR-VT: Handwritten Text Recognition with Vision Transformer [7.997204893256558]
  We explore the application of Vision Transformer (ViT) for handwritten text recognition. Previous transformer-based models required external data or extensive pre-training on large datasets to excel. We find that incorporating a ConAwareal Neural Network (CNN) for feature extraction instead of the original patch embedding and employ Sharpness Minimization (SAM) encoder ensures that the model can converge towards flatter minima.
  arXiv  Detail & Related papers  (2024-09-13T06:46:23Z)
• Optimizing Vision Transformers with Data-Free Knowledge Transfer [8.323741354066474]
  Vision transformers (ViTs) have excelled in various computer vision tasks due to their superior ability to capture long-distance dependencies. We propose compressing large ViT models using Knowledge Distillation (KD), which is implemented data-free to circumvent limitations related to data availability.
  arXiv  Detail & Related papers  (2024-08-12T07:03:35Z)
• Rethinking Hierarchicies in Pre-trained Plain Vision Transformer [76.35955924137986]
  Self-supervised pre-training vision transformer (ViT) via masked image modeling (MIM) has been proven very effective. customized algorithms should be carefully designed for the hierarchical ViTs, e.g., GreenMIM, instead of using the vanilla and simple MAE for the plain ViT. This paper proposes a novel idea of disentangling the hierarchical architecture design from the self-supervised pre-training.
  arXiv  Detail & Related papers  (2022-11-03T13:19:23Z)
• How to Train Vision Transformer on Small-scale Datasets? [4.56717163175988]
  In contrast to convolutional neural networks, Vision Transformer lacks inherent inductive biases. We show that self-supervised inductive biases can be learned directly from small-scale datasets. This allows to train these models without large-scale pre-training, changes to model architecture or loss functions.
  arXiv  Detail & Related papers  (2022-10-13T17:59:19Z)
• Improving Vision Transformers by Revisiting High-frequency Components [106.7140968644414]
  We show that Vision Transformer (ViT) models are less effective in capturing the high-frequency components of images than CNN models. To compensate, we propose HAT, which directly augments high-frequency components of images via adversarial training. We show that HAT can consistently boost the performance of various ViT models.
  arXiv  Detail & Related papers  (2022-04-03T05:16:51Z)
• Scaling Vision Transformers [82.08465256393514]
  We study how Vision Transformers scale and characterize the relationships between error rate, data, and compute. We train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90.45% top-1 accuracy. The model also performs well on few-shot learning, for example, attaining 84.86% top-1 accuracy on ImageNet with only 10 examples per class.
  arXiv  Detail & Related papers  (2021-06-08T17:47:39Z)
• When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations [111.44860506703307]
  Vision Transformers (ViTs) and existing VisionNets signal efforts on replacing hand-wired features or inductive throughputs with general-purpose neural architectures. This paper investigates ViTs and Res-Mixers from the lens of loss geometry, intending to improve the models' data efficiency at training and inference. We show that the improved robustness attributes to sparser active neurons in the first few layers. The resultant ViTs outperform Nets of similar size and smoothness when trained from scratch on ImageNet without large-scale pretraining or strong data augmentations.
  arXiv  Detail & Related papers  (2021-06-03T02:08:03Z)
• Visformer: The Vision-friendly Transformer [105.52122194322592]
  We propose a new architecture named Visformer, which is abbreviated from the Vision-friendly Transformer' With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy.
  arXiv  Detail & Related papers  (2021-04-26T13:13:03Z)
• DeepViT: Towards Deeper Vision Transformer [92.04063170357426]
  Vision transformers (ViTs) have been successfully applied in image classification tasks recently. We show that, unlike convolution neural networks (CNNs)that can be improved by stacking more convolutional layers, the performance of ViTs saturate fast when scaled to be deeper. We propose a simple yet effective method, named Re-attention, to re-generate the attention maps to increase their diversity.
  arXiv  Detail & Related papers  (2021-03-22T14:32:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.