Token Turing Machines are Efficient Vision Models

• URL: http://arxiv.org/abs/2409.07613v1
• Date: Wed, 11 Sep 2024 20:50:41 GMT
• Title: Token Turing Machines are Efficient Vision Models
• Abstract summary: We propose Vision Token Turing Machines (ViTTM), an efficient, low-latency, memory-augmented Vision Transformer (ViT) ViTTMs are designed for non-sequential computer vision tasks such as image classification and segmentation.
• Score: 2.8674240660437778
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose Vision Token Turing Machines (ViTTM), an efficient, low-latency, memory-augmented Vision Transformer (ViT). Our approach builds on Neural Turing Machines and Token Turing Machines, which were applied to NLP and sequential visual understanding tasks. ViTTMs are designed for non-sequential computer vision tasks such as image classification and segmentation. Our model creates two sets of tokens: process tokens and memory tokens; process tokens pass through encoder blocks and read-write from memory tokens at each encoder block in the network, allowing them to store and retrieve information from memory. By ensuring that there are fewer process tokens than memory tokens, we are able to reduce the inference time of the network while maintaining its accuracy. On ImageNet-1K, the state-of-the-art ViT-B has median latency of 529.5ms and 81.0% accuracy, while our ViTTM-B is 56% faster (234.1ms), with 2.4 times fewer FLOPs, with an accuracy of 82.9%. On ADE20K semantic segmentation, ViT-B achieves 45.65mIoU at 13.8 frame-per-second (FPS) whereas our ViTTM-B model acheives a 45.17 mIoU with 26.8 FPS (+94%).

Related papers

• Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding [51.91097761028129]
  We introduce ProVideLLM, an end-to-end framework for real-time procedural video understanding. ProVideLLM integrates a multimodal cache configured to store two types of tokens. By interleaving these tokens in our multimodal cache, ProVideLLM ensures sub-linear scaling of memory and compute with video length.
  arXiv  Detail & Related papers  (2025-04-10T17:13:08Z)
• OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference [9.78611123915888]
  We present OmniVLM, a vision-language model for efficient on-device inference. It reduces visual token sequence length from 729 to 81 tokens, significantly reducing computational overhead. It outperforms existing baselines like nanoLLAVA within a 968M- parameter footprint.
  arXiv  Detail & Related papers  (2024-12-16T06:38:00Z)
• Token Pruning using a Lightweight Background Aware Vision Transformer [0.6856888934092934]
  Token pruning reduces the number of input tokens to the ViT based on importance criteria of each token. Background tokens can be pruned completely or partially before feeding to a ViT based object detector. We show a 2 layer BAViT-small model as pre-processor to YOLOS can increase the throughput by 30% - 40% with a mAP drop of 3% without any sparse fine-tuning.
  arXiv  Detail & Related papers  (2024-10-12T01:44:54Z)
• Dynamic neural network with memristive CIM and CAM for 2D and 3D vision [57.6208980140268]
  We propose a semantic memory-based dynamic neural network (DNN) using memristor. The network associates incoming data with the past experience stored as semantic vectors. We validate our co-designs, using a 40nm memristor macro, on ResNet and PointNet++ for classifying images and 3D points from the MNIST and ModelNet datasets.
  arXiv  Detail & Related papers  (2024-07-12T04:55:57Z)
• ViT-1.58b: Mobile Vision Transformers in the 1-bit Era [27.74281483852495]
  This paper introduces ViT-1.58b, a novel 1.58-bit quantized ViT model designed to drastically reduce memory and computational overhead. Experiments on CIFAR-10 and ImageNet-1k demonstrate that ViT-1.58b maintains comparable accuracy to full-precision Vit.
  arXiv  Detail & Related papers  (2024-06-26T04:01:19Z)
• Accelerating Transformers with Spectrum-Preserving Token Merging [43.463808781808645]
  PiToMe prioritizes the preservation of informative tokens using an additional metric termed the energy score. Experimental findings demonstrate that PiToMe saved from 40-60% FLOPs of the base models.
  arXiv  Detail & Related papers  (2024-05-25T09:37:01Z)
• AiluRus: A Scalable ViT Framework for Dense Prediction [95.1313839257891]
  Vision transformers (ViTs) have emerged as a prevalent architecture for vision tasks owing to their impressive performance. We propose to apply adaptive resolution for different regions in the image according to their importance. We evaluate our proposed method on three different datasets and observe promising performance.
  arXiv  Detail & Related papers  (2023-11-02T12:48:43Z)
• Token Turing Machines [53.22971546637947]
  Token Turing Machines (TTM) is a sequential, autoregressive Transformer model with memory for real-world sequential visual understanding. Our model is inspired by the seminal Neural Turing Machine, and has an external memory consisting of a set of tokens which summarise the previous history.
  arXiv  Detail & Related papers  (2022-11-16T18:59:18Z)
• DaViT: Dual Attention Vision Transformers [94.62855697081079]
  We introduce Dual Attention Vision Transformers (DaViT) DaViT is a vision transformer architecture that is able to capture global context while maintaining computational efficiency. We show our DaViT achieves state-of-the-art performance on four different tasks with efficient computations.
  arXiv  Detail & Related papers  (2022-04-07T17:59:32Z)
• SepViT: Separable Vision Transformer [20.403430632658946]
  Vision Transformers often rely on extensive computational costs to achieve high performance, which is burdensome to deploy on resource-constrained devices. We draw lessons from depthwise separable convolution and imitate its ideology to design an efficient Transformer backbone, i.e., Separable Vision Transformer, abbreviated as SepViT. SepViT helps to carry out the local-global information interaction within and among the windows in sequential order via a depthwise separable self-attention.
  arXiv  Detail & Related papers  (2022-03-29T09:20:01Z)
• Self-slimmed Vision Transformer [52.67243496139175]
  Vision transformers (ViTs) have become the popular structures and outperformed convolutional neural networks (CNNs) on various vision tasks. We propose a generic self-slimmed learning approach for vanilla ViTs, namely SiT. Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs.
  arXiv  Detail & Related papers  (2021-11-24T16:48:57Z)
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet [128.96032932640364]
  We propose a new Tokens-To-Token Vision Transformers (T2T-ViT) to solve vision tasks. T2T-ViT reduces the parameter counts and MACs of vanilla ViT by 200%, while achieving more than 2.5% improvement when trained from scratch on ImageNet. For example, T2T-ViT with ResNet50 comparable size can achieve 80.7% top-1 accuracy on ImageNet.
  arXiv  Detail & Related papers  (2021-01-28T13:25:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.