Vision Transformer attention alignment with human visual perception in aesthetic object evaluation

• URL: http://arxiv.org/abs/2507.17616v1
• Date: Wed, 23 Jul 2025 15:47:34 GMT
• Title: Vision Transformer attention alignment with human visual perception in aesthetic object evaluation
• Authors: Miguel Carrasco, César González-Martín, José Aranda, Luis Oliveros,
• Abstract summary: Visual attention mechanisms play a crucial role in human perception and aesthetic evaluation.<n>Recent advances in Vision Transformers (ViTs) have demonstrated remarkable capabilities in computer vision tasks.<n>This study investigates the correlation between human visual attention and ViT attention mechanisms when evaluating handcrafted objects.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Visual attention mechanisms play a crucial role in human perception and aesthetic evaluation. Recent advances in Vision Transformers (ViTs) have demonstrated remarkable capabilities in computer vision tasks, yet their alignment with human visual attention patterns remains underexplored, particularly in aesthetic contexts. This study investigates the correlation between human visual attention and ViT attention mechanisms when evaluating handcrafted objects. We conducted an eye-tracking experiment with 30 participants (9 female, 21 male, mean age 24.6 years) who viewed 20 artisanal objects comprising basketry bags and ginger jars. Using a Pupil Labs eye-tracker, we recorded gaze patterns and generated heat maps representing human visual attention. Simultaneously, we analyzed the same objects using a pre-trained ViT model with DINO (Self-DIstillation with NO Labels), extracting attention maps from each of the 12 attention heads. We compared human and ViT attention distributions using Kullback-Leibler divergence across varying Gaussian parameters (sigma=0.1 to 3.0). Statistical analysis revealed optimal correlation at sigma=2.4 +-0.03, with attention head #12 showing the strongest alignment with human visual patterns. Significant differences were found between attention heads, with heads #7 and #9 demonstrating the greatest divergence from human attention (p< 0.05, Tukey HSD test). Results indicate that while ViTs exhibit more global attention patterns compared to human focal attention, certain attention heads can approximate human visual behavior, particularly for specific object features like buckles in basketry items. These findings suggest potential applications of ViT attention mechanisms in product design and aesthetic evaluation, while highlighting fundamental differences in attention strategies between human perception and current AI models.

Related papers

• Visual Attention Graph [21.860357478331107]
  We propose a new attention representation, called Attention Graph, to simultaneously code the visual saliency and scanpath.<n>In the attention graph, the semantic-based scanpath is defined by the path on the graph, while saliency of objects can be obtained by computing fixation density on each node.
  arXiv  Detail & Related papers  (2025-03-11T15:22:44Z)
• Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models [51.58859621164201]
  We introduce visual stimuli and human judgments on visual cognition tasks to evaluate performance across cognitive domains.<n>We fine-tune models on ground truth data for intuitive physics and causal reasoning.<n>We find that task-specific fine-tuning does not contribute to robust human-like generalization to data with other visual characteristics.
  arXiv  Detail & Related papers  (2025-02-21T18:58:30Z)
• Human Gaze Boosts Object-Centered Representation Learning [7.473473243713322]
  Recent self-supervised learning models trained on human-like egocentric visual inputs substantially underperform on image recognition tasks compared to humans.<n>Here, we investigate whether focusing on central visual information boosts egocentric visual object learning.<n>Our experiments demonstrate that focusing on central vision leads to better object-centered representations.
  arXiv  Detail & Related papers  (2025-01-06T12:21:40Z)
• Evaluating Multiview Object Consistency in Humans and Image Models [68.36073530804296]
  We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape. We collect 35K trials of behavioral data from over 500 participants. We then evaluate the performance of common vision models.
  arXiv  Detail & Related papers  (2024-09-09T17:59:13Z)
• Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition [13.956664101032006]
  We first collect a novel gaze fixation dataset named IG, comprising 530,000 fixation points across 740 diverse interaction categories. We then introduce the zero-shot interaction-oriented attention prediction task ZeroIA, which challenges models to predict visual cues for interactions not encountered during training. Thirdly, we present the Interactive Attention model IA, designed to emulate human observers cognitive processes to tackle the ZeroIA problem.
  arXiv  Detail & Related papers  (2024-05-16T09:34:57Z)
• Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models [56.257840490146]
  ConCue is a novel approach for improving visual feature extraction in HOI detection. We develop a transformer-based feature extraction module with a multi-tower architecture that integrates contextual cues into both instance and interaction detectors.
  arXiv  Detail & Related papers  (2023-11-26T09:11:32Z)
• BI AVAN: Brain inspired Adversarial Visual Attention Network [67.05560966998559]
  We propose a brain-inspired adversarial visual attention network (BI-AVAN) to characterize human visual attention directly from functional brain activity. Our model imitates the biased competition process between attention-related/neglected objects to identify and locate the visual objects in a movie frame the human brain focuses on in an unsupervised manner.
  arXiv  Detail & Related papers  (2022-10-27T22:20:36Z)
• Guiding Visual Attention in Deep Convolutional Neural Networks Based on Human Eye Movements [0.0]
  Deep Convolutional Neural Networks (DCNNs) were originally inspired by principles of biological vision. Recent advances in deep learning seem to decrease this similarity. We investigate a purely data-driven approach to obtain useful models.
  arXiv  Detail & Related papers  (2022-06-21T17:59:23Z)
• Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze? [0.0]
  Self-attention functions in state-of-the-art NLP models often correlate with human attention. We investigate whether self-attention in large-scale pre-trained language models is as predictive of human eye fixation patterns during task-reading as classical cognitive models of human attention.
  arXiv  Detail & Related papers  (2022-04-25T08:23:13Z)
• Attention Mechanisms in Computer Vision: A Survey [75.6074182122423]
  We provide a comprehensive review of various attention mechanisms in computer vision. We categorize them according to approach, such as channel attention, spatial attention, temporal attention and branch attention. We suggest future directions for attention mechanism research.
  arXiv  Detail & Related papers  (2021-11-15T09:18:40Z)
• Gaze Perception in Humans and CNN-Based Model [66.89451296340809]
  We compare how a CNN (convolutional neural network) based model of gaze and humans infer the locus of attention in images of real-world scenes. We show that compared to the model, humans' estimates of the locus of attention are more influenced by the context of the scene.
  arXiv  Detail & Related papers  (2021-04-17T04:52:46Z)
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions [50.435861435121915]
  We use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. Our experiments show that our "muscly-supervised" representation outperforms a visual-only state-of-the-art method MoCo.
  arXiv  Detail & Related papers  (2020-10-16T17:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.