Unified Local and Global Attention Interaction Modeling for Vision Transformers

• URL: http://arxiv.org/abs/2412.18778v1
• Date: Wed, 25 Dec 2024 04:53:19 GMT
• Title: Unified Local and Global Attention Interaction Modeling for Vision Transformers
• Authors: Tan Nguyen, Coy D. Heldermon, Corey Toler-Franklin,
• Abstract summary: We present a novel method that extends the self-attention mechanism of a vision transformer (ViT) for more accurate object detection across diverse datasets. ViTs show strong capability for image understanding tasks such as object detection, segmentation, and classification. We introduce two modifications to the traditional self-attention framework; a novel aggressive convolution pooling strategy for local feature mixing, and a new conceptual attention transformation.
• Score: 1.9571946424055506
• License:
• Abstract: We present a novel method that extends the self-attention mechanism of a vision transformer (ViT) for more accurate object detection across diverse datasets. ViTs show strong capability for image understanding tasks such as object detection, segmentation, and classification. This is due in part to their ability to leverage global information from interactions among visual tokens. However, the self-attention mechanism in ViTs are limited because they do not allow visual tokens to exchange local or global information with neighboring features before computing global attention. This is problematic because tokens are treated in isolation when attending (matching) to other tokens, and valuable spatial relationships are overlooked. This isolation is further compounded by dot-product similarity operations that make tokens from different semantic classes appear visually similar. To address these limitations, we introduce two modifications to the traditional self-attention framework; a novel aggressive convolution pooling strategy for local feature mixing, and a new conceptual attention transformation to facilitate interaction and feature exchange between semantic concepts. Experimental results demonstrate that local and global information exchange among visual features before self-attention significantly improves performance on challenging object detection tasks and generalizes across multiple benchmark datasets and challenging medical datasets. We publish source code and a novel dataset of cancerous tumors (chimeric cell clusters).

Related papers

• KNN Transformer with Pyramid Prompts for Few-Shot Learning [52.735070934075736]
  Few-Shot Learning aims to recognize new classes with limited labeled data. Recent studies have attempted to address the challenge of rare samples with textual prompts to modulate visual features.
  arXiv  Detail & Related papers  (2024-10-14T07:39:30Z)
• Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detection [57.883265488038134]
  We propose a hierarchical graph interaction network termed HGINet for camouflaged object detection. The network is capable of discovering imperceptible objects via effective graph interaction among the hierarchical tokenized features. Our experiments demonstrate the superior performance of HGINet compared to existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-27T12:53:25Z)
• Other Tokens Matter: Exploring Global and Local Features of Vision Transformers for Object Re-Identification [63.147482497821166]
  We first explore the influence of global and local features of ViT and then propose a novel Global-Local Transformer (GLTrans) for high-performance object Re-ID. Our proposed method achieves superior performance on four object Re-ID benchmarks.
  arXiv  Detail & Related papers  (2024-04-23T12:42:07Z)
• Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model [3.3772986620114387]
  We introduce ISA-HOI, which extensively leverages knowledge from CLIP, aligning interactive semantics between visual and textual features. Our method achieves competitive results on the HICO-DET and V-COCO benchmarks with much fewer training epochs, and outperforms the state-of-the-art under zero-shot settings.
  arXiv  Detail & Related papers  (2024-04-19T07:24:32Z)
• Dissecting Query-Key Interaction in Vision Transformers [4.743574336827573]
  Self-attention in vision transformers is often thought to perform perceptual grouping. We analyze the query-key interaction by the singular value decomposition of the interaction matrix.
  arXiv  Detail & Related papers  (2024-04-04T20:06:07Z)
• Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection [70.96299509159981]
  Human-Object Interaction (HOI) detection is a core task for human-centric image understanding. Recent one-stage methods adopt a transformer decoder to collect image-wide cues that are useful for interaction prediction. Traditional two-stage methods benefit significantly from their ability to compose interaction features in a disentangled and explainable manner.
  arXiv  Detail & Related papers  (2023-12-04T08:02:59Z)
• Masked Momentum Contrastive Learning for Zero-shot Semantic Understanding [39.424931953675994]
  Self-supervised pretraining (SSP) has emerged as a popular technique in machine learning, enabling the extraction of meaningful feature representations without labelled data. This study endeavours to evaluate the effectiveness of pure self-supervised learning (SSL) techniques in computer vision tasks.
  arXiv  Detail & Related papers  (2023-08-22T13:55:57Z)
• AttentionViz: A Global View of Transformer Attention [60.82904477362676]
  We present a new visualization technique designed to help researchers understand the self-attention mechanism in transformers. The main idea behind our method is to visualize a joint embedding of the query and key vectors used by transformer models to compute attention. We create an interactive visualization tool, AttentionViz, based on these joint query-key embeddings.
  arXiv  Detail & Related papers  (2023-05-04T23:46:49Z)
• Rethinking Query-Key Pairwise Interactions in Vision Transformers [5.141895475956681]
  We propose key-only attention, which excludes query-key pairwise interactions and uses a compute-efficient saliency-gate to obtain attention weights. We develop a new self-attention model family, LinGlos, which reach state-of-the-art accuracies on the parameter-limited setting of ImageNet classification benchmark.
  arXiv  Detail & Related papers  (2022-07-01T03:36:49Z)
• Visualizing and Understanding Patch Interactions in Vision Transformer [96.70401478061076]
  Vision Transformer (ViT) has become a leading tool in various computer vision tasks. We propose a novel explainable visualization approach to analyze and interpret the crucial attention interactions among patches for vision transformer.
  arXiv  Detail & Related papers  (2022-03-11T13:48:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.