A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Maps

• URL: http://arxiv.org/abs/2603.09613v2
• Date: Wed, 11 Mar 2026 12:40:13 GMT
• Title: A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Maps
• Abstract summary: We draw inspiration from the human visual system to create smarter image processing models.<n>Using DINO, a self-supervised Vision Transformer, we explore a saccade inspired method to focus the processing of information on key regions in visual space.<n>This selective-processing strategy preserves most of the full-image classification performance and can even outperform it in certain cases.
• Score: 0.9332987715848716
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human vision achieves remarkable perceptual performance while operating under strict metabolic constraints. A key ingredient is the selective attention mechanism, driven by rapid saccadic eye movements that constantly reposition the high-resolution fovea onto task-relevant locations, unlike conventional AI systems that process entire images with equal emphasis. Our work aims to draw inspiration from the human visual system to create smarter, more efficient image processing models. Using DINO, a self-supervised Vision Transformer that produces attention maps strikingly similar to human gaze patterns, we explore a saccade inspired method to focus the processing of information on key regions in visual space. To do so, we use the ImageNet dataset in a standard classification task and measure how each successive saccade affects the model's class scores. This selective-processing strategy preserves most of the full-image classification performance and can even outperform it in certain cases. By benchmarking against established saliency models built for human gaze prediction, we demonstrate that DINO provides superior fixation guidance for selecting informative regions. These findings highlight Vision Transformer attention as a promising basis for biologically inspired active vision and open new directions for efficient, neuromorphic visual processing.

Related papers

• STARE: Predicting Decision Making Based on Spatio-Temporal Eye Movements [49.906485205551746]
  The present proposes a Deep Learning architecture for the prediction of various consumer choice behaviors from time series of raw gaze or eye fixations on images of the decision environment.<n>We compare STARE with several state-of-the art alternatives on multiple datasets with the purpose of predicting consumer choice behaviors from eye movements.
  arXiv  Detail & Related papers  (2025-08-06T07:20:31Z)
• Human Scanpath Prediction in Target-Present Visual Search with Semantic-Foveal Bayesian Attention [49.99728312519117]
  SemBA-FAST is a top-down framework designed for predicting human visual attention in target-present visual search.<n>We evaluate SemBA-FAST on the COCO-Search18 benchmark dataset, comparing its performance against other scanpath prediction models.<n>These findings provide valuable insights into the capabilities of semantic-foveal probabilistic frameworks for human-like attention modelling.
  arXiv  Detail & Related papers  (2025-07-24T15:19:23Z)
• Align and Surpass Human Camouflaged Perception: Visual Refocus Reinforcement Fine-Tuning [18.13538667261998]
  Current multi-modal models exhibit a notable misalignment with the human visual system when identifying objects that are visually assimilated into the background.<n>We build a visual system that mimicks human visual camouflaged perception to progressively and iteratively refocus' visual concealed content.
  arXiv  Detail & Related papers  (2025-05-26T07:27:18Z)
• Convolution goes higher-order: a biologically inspired mechanism empowers image classification [0.8999666725996975]
  We propose a novel approach to image classification inspired by complex nonlinear biological visual processing.<n>Our model incorporates a Volterra-like expansion of the convolution operator, capturing multiplicative interactions.<n>Our work bridges neuroscience and deep learning, offering a path towards more effective, biologically inspired computer vision models.
  arXiv  Detail & Related papers  (2024-12-09T18:33:09Z)
• Vision Eagle Attention: a new lens for advancing image classification [0.8158530638728501]
  I introduce Vision Eagle Attention, a novel attention mechanism that enhances visual feature extraction using convolutional spatial attention.<n>The model applies convolution to capture local spatial features and generates an attention map that selectively emphasizes the most informative regions of the image.<n>I have integrated Vision Eagle Attention into a lightweight ResNet-18 architecture, demonstrating that this combination results in an efficient and powerful model.
  arXiv  Detail & Related papers  (2024-11-15T20:21:59Z)
• InstructDiffusion: A Generalist Modeling Interface for Vision Tasks [52.981128371910266]
  We present InstructDiffusion, a framework for aligning computer vision tasks with human instructions. InstructDiffusion could handle a variety of vision tasks, including understanding tasks and generative tasks. It even exhibits the ability to handle unseen tasks and outperforms prior methods on novel datasets.
  arXiv  Detail & Related papers  (2023-09-07T17:56:57Z)
• Top-Down Visual Attention from Analysis by Synthesis [87.47527557366593]
  We consider top-down attention from a classic Analysis-by-Synthesis (AbS) perspective of vision. We propose Analysis-by-Synthesis Vision Transformer (AbSViT), which is a top-down modulated ViT model that variationally approximates AbS, and controllable achieves top-down attention.
  arXiv  Detail & Related papers  (2023-03-23T05:17:05Z)
• A domain adaptive deep learning solution for scanpath prediction of paintings [66.46953851227454]
  This paper focuses on the eye-movement analysis of viewers during the visual experience of a certain number of paintings. We introduce a new approach to predicting human visual attention, which impacts several cognitive functions for humans. The proposed new architecture ingests images and returns scanpaths, a sequence of points featuring a high likelihood of catching viewers' attention.
  arXiv  Detail & Related papers  (2022-09-22T22:27:08Z)
• Peripheral Vision Transformer [52.55309200601883]
  We take a biologically inspired approach and explore to model peripheral vision in deep neural networks for visual recognition. We propose to incorporate peripheral position encoding to the multi-head self-attention layers to let the network learn to partition the visual field into diverse peripheral regions given training data. We evaluate the proposed network, dubbed PerViT, on the large-scale ImageNet dataset and systematically investigate the inner workings of the model for machine perception.
  arXiv  Detail & Related papers  (2022-06-14T12:47:47Z)
• Behind the Machine's Gaze: Biologically Constrained Neural Networks Exhibit Human-like Visual Attention [40.878963450471026]
  We propose the Neural Visual Attention (NeVA) algorithm to generate visual scanpaths in a top-down manner. We show that the proposed method outperforms state-of-the-art unsupervised human attention models in terms of similarity to human scanpaths.
  arXiv  Detail & Related papers  (2022-04-19T18:57:47Z)
• ViCE: Self-Supervised Visual Concept Embeddings as Contextual and Pixel Appearance Invariant Semantic Representations [77.3590853897664]
  This work presents a self-supervised method to learn dense semantically rich visual embeddings for images inspired by methods for learning word embeddings in NLP.
  arXiv  Detail & Related papers  (2021-11-24T12:27:30Z)
• Exploring Vision Transformers for Fine-grained Classification [0.0]
  We propose a multi-stage ViT framework for fine-grained image classification tasks, which localizes the informative image regions without requiring architectural changes. We demonstrate the value of our approach by experimenting with four popular fine-grained benchmarks: CUB-200-2011, Stanford Cars, Stanford Dogs, and FGVC7 Plant Pathology.
  arXiv  Detail & Related papers  (2021-06-19T23:57:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.