What They Saw, Not Just Where They Looked: Semantic Scanpath Similarity via VLMs and NLP metric

• URL: http://arxiv.org/abs/2604.08494v1
• Date: Thu, 09 Apr 2026 17:36:22 GMT
• Title: What They Saw, Not Just Where They Looked: Semantic Scanpath Similarity via VLMs and NLP metric
• Abstract summary: We present a semantic scanpath similarity framework that integrates vision-language models into eye-tracking analysis.<n>Semantic similarity is then computed using embedding-based and lexical NLP metrics.<n>Experiments on free-viewing eye-tracking data demonstrate that semantic similarity captures partially independent variance from geometric alignment.
• Score: 50.699179933343686
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Scanpath similarity metrics are central to eye-movement research, yet existing methods predominantly evaluate spatial and temporal alignment while neglecting semantic equivalence between attended image regions. We present a semantic scanpath similarity framework that integrates vision-language models (VLMs) into eye-tracking analysis. Each fixation is encoded under controlled visual context (patch-based and marker-based strategies) and transformed into concise textual descriptions, which are aggregated into scanpath-level representations. Semantic similarity is then computed using embedding-based and lexical NLP metrics and compared against established spatial measures, including MultiMatch and DTW. Experiments on free-viewing eye-tracking data demonstrate that semantic similarity captures partially independent variance from geometric alignment, revealing cases of high content agreement despite spatial divergence. We further analyze the impact of contextual encoding on description fidelity and metric stability. Our findings suggest that multimodal foundation models enable interpretable, content-aware extensions of classical scanpath analysis, providing a complementary dimension for gaze research within the ETRA community.

Related papers

• MAPS: Multi-Anchor Projection Similarity for Joint Vision-Language Geo-Localization [14.254687625765547]
  We formulate vision-language geo-localization with joint image-text queries as a multi-anchor geometric alignment problem.<n>New metric Multi-Anchor Projection Similarity (MAPS) measures similarity by the projection length of the target feature onto this plane.<n>Proposed framework, similarity metric, and training objective jointly yield state-of-the-art performance in vision-language geo-localization.
  arXiv  Detail & Related papers  (2026-06-21T15:01:53Z)
• Seeking Consensus: Geometric-Semantic On-the-Fly Recalibration for Open-Vocabulary Remote Sensing Semantic Segmentation [27.476148859056114]
  SeeCo is a plug-and-play framework to boost the performance of training-free OVSS models in remote sensing images.<n>It recalibrates arbitrary OVSS models on-the-fly by seeking dual consensus.<n>Experiments on eight remote sensing OVSS benchmarks show consistent gains.
  arXiv  Detail & Related papers  (2026-04-29T01:57:21Z)
• Characterizing Human Semantic Navigation in Concept Production as Trajectories in Embedding Space [0.0]
  We introduce a framework that represents concept production as navigation through embedding space.<n>We construct participant-specific semantic trajectories based on cumulative embeddings and extract geometric and dynamical metrics.<n>We evaluate the framework on four datasets across different languages, spanning different property generation tasks.
  arXiv  Detail & Related papers  (2026-02-05T18:23:04Z)
• Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds [49.95082206008502]
  Alignment across Trees is a method that constructs and aligns tree-like hierarchical features for both image and text modalities.<n>We introduce a semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers.
  arXiv  Detail & Related papers  (2025-10-31T11:32:15Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• HyperPath: Knowledge-Guided Hyperbolic Semantic Hierarchy Modeling for WSI Analysis [21.380034877048644]
  We propose HyperPath, a novel method that integrates knowledge from textual descriptions to guide the modeling of semantic hierarchies in hyperbolic space.<n>Our approach adapts both visual and textual features extracted by pathology vision-language foundation models to the hyperbolic space.<n>Our method achieves superior performance across tasks compared to existing methods, highlighting the potential of hyperbolic embeddings for WSI analysis.
  arXiv  Detail & Related papers  (2025-06-19T15:30:33Z)
• Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment [25.209622555403527]
  We propose a framework called Asymmetric Visual Semantic Embedding (AVSE) to dynamically select features from various regions of images tailored to different textual inputs for similarity calculation.<n>AVSE calculates visual semantic similarity by finding the optimal match of meta-semantic embeddings of two modalities.<n>Our proposed AVSE model is extensively evaluated on the large-scale MS-COCO and Flickr30K datasets.
  arXiv  Detail & Related papers  (2025-03-10T06:38:41Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• Image Synthesis via Semantic Composition [74.68191130898805]
  We present a novel approach to synthesize realistic images based on their semantic layouts. It hypothesizes that for objects with similar appearance, they share similar representation. Our method establishes dependencies between regions according to their appearance correlation, yielding both spatially variant and associated representations.
  arXiv  Detail & Related papers  (2021-09-15T02:26:07Z)
• Similarity Reasoning and Filtration for Image-Text Matching [85.68854427456249]
  We propose a novel Similarity Graph Reasoning and Attention filtration network for image-text matching. Similarity Graph Reasoning (SGR) module relying on one graph convolutional neural network is introduced to infer relation-aware similarities with both the local and global alignments. We demonstrate the superiority of the proposed method with achieving state-of-the-art performances on the Flickr30K and MSCOCO datasets.
  arXiv  Detail & Related papers  (2021-01-05T06:29:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.