MicroViT: A Vision Transformer with Low Complexity Self Attention for Edge Device

• URL: http://arxiv.org/abs/2502.05800v1
• Date: Sun, 09 Feb 2025 08:04:39 GMT
• Title: MicroViT: A Vision Transformer with Low Complexity Self Attention for Edge Device
• Authors: Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu, Wen-Kai Kuo, Jun-Wei Hsieh,
• Abstract summary: Vision Transformer (ViT) has demonstrated state-of-the-art performance in various computer vision tasks, but its high computational demands make it impractical for edge devices with limited resources. This paper presents MicroViT, a lightweight Vision Transformer architecture optimized for edge devices by significantly reducing computational complexity while maintaining high accuracy.
• Score: 3.617580194719686
• License:
• Abstract: The Vision Transformer (ViT) has demonstrated state-of-the-art performance in various computer vision tasks, but its high computational demands make it impractical for edge devices with limited resources. This paper presents MicroViT, a lightweight Vision Transformer architecture optimized for edge devices by significantly reducing computational complexity while maintaining high accuracy. The core of MicroViT is the Efficient Single Head Attention (ESHA) mechanism, which utilizes group convolution to reduce feature redundancy and processes only a fraction of the channels, thus lowering the burden of the self-attention mechanism. MicroViT is designed using a multi-stage MetaFormer architecture, stacking multiple MicroViT encoders to enhance efficiency and performance. Comprehensive experiments on the ImageNet-1K and COCO datasets demonstrate that MicroViT achieves competitive accuracy while significantly improving 3.6 faster inference speed and reducing energy consumption with 40% higher efficiency than the MobileViT series, making it suitable for deployment in resource-constrained environments such as mobile and edge devices.

Related papers

• CHOSEN: Compilation to Hardware Optimization Stack for Efficient Vision Transformer Inference [4.523939613157408]
  Vision Transformers (ViTs) represent a groundbreaking shift in machine learning approaches to computer vision. This paper introduces CHOSEN, a software-hardware co-design framework to address these challenges and offer an automated framework for ViT deployment on the FPGAs. ChoSEN achieves a 1.5x and 1.42x improvement in the throughput on the DeiT-S and DeiT-B models.
  arXiv  Detail & Related papers  (2024-07-17T16:56:06Z)
• DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices [42.89175608336226]
  Vision transformer (ViT) has achieved state-of-the-art performance on multiple computer vision benchmarks. ViT models suffer from vast amounts of parameters and high computation cost, leading to difficult deployment on resource-constrained edge devices. We propose a collaborative inference framework termed DeViT to facilitate edge deployment by decomposing large ViTs.
  arXiv  Detail & Related papers  (2023-09-10T12:26:17Z)
• Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts [60.1586169973792]
  M$3$ViT is the latest multi-task ViT model that introduces mixture-of-experts (MoE) MoE achieves better accuracy and over 80% reduction computation but leaves challenges for efficient deployment on FPGA. Our work, dubbed Edge-MoE, solves the challenges to introduce the first end-to-end FPGA accelerator for multi-task ViT with a collection of architectural innovations.
  arXiv  Detail & Related papers  (2023-05-30T02:24:03Z)
• Fast GraspNeXt: A Fast Self-Attention Neural Network Architecture for Multi-task Learning in Computer Vision Tasks for Robotic Grasping on the Edge [80.88063189896718]
  High architectural and computational complexity can result in poor suitability for deployment on embedded devices. Fast GraspNeXt is a fast self-attention neural network architecture tailored for embedded multi-task learning in computer vision tasks for robotic grasping.
  arXiv  Detail & Related papers  (2023-04-21T18:07:14Z)
• Rethinking Vision Transformers for MobileNet Size and Speed [58.01406896628446]
  We propose a novel supernet with low latency and high parameter efficiency. We also introduce a novel fine-grained joint search strategy for transformer models. This work demonstrate that properly designed and optimized vision transformers can achieve high performance even with MobileNet-level size and speed.
  arXiv  Detail & Related papers  (2022-12-15T18:59:12Z)
• ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention [23.874485033096917]
  Vision Transformer (ViT) has emerged as a competitive alternative to convolutional neural networks for various computer vision applications. We propose a first-of-its-kind algorithm- hardware codesigned framework, dubbed ViTALiTy, for boosting the inference efficiency of ViTs. ViTALiTy unifies both low-rank and sparse components of the attention in ViTs.
  arXiv  Detail & Related papers  (2022-11-09T18:58:21Z)
• EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers [88.52500757894119]
  Self-attention based vision transformers (ViTs) have emerged as a very competitive architecture alternative to convolutional neural networks (CNNs) in computer vision. We introduce EdgeViTs, a new family of light-weight ViTs that, for the first time, enable attention-based vision models to compete with the best light-weight CNNs.
  arXiv  Detail & Related papers  (2022-05-06T18:17:19Z)
• Coarse-to-Fine Vision Transformer [83.45020063642235]
  We propose a coarse-to-fine vision transformer (CF-ViT) to relieve computational burden while retaining performance. Our proposed CF-ViT is motivated by two important observations in modern ViT models. Our CF-ViT reduces 53% FLOPs of LV-ViT, and also achieves 2.01x throughput.
  arXiv  Detail & Related papers  (2022-03-08T02:57:49Z)
• AdaViT: Adaptive Tokens for Efficient Vision Transformer [91.88404546243113]
  We introduce AdaViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. AdaViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
  arXiv  Detail & Related papers  (2021-12-14T18:56:07Z)
• HRViT: Multi-Scale High-Resolution Vision Transformer [19.751569057142806]
  Vision transformers (ViTs) have attracted much attention for their superior performance on computer vision tasks. We present an efficient integration of high-resolution multi-branch architectures with vision transformers, dubbed HRViT. The proposed HRViT achieves 50.20% mIoU on ADE20K and 83.16% mIoU on Cityscapes for semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-11-01T19:49:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.