Decoding Reading Goals from Eye Movements

• URL: http://arxiv.org/abs/2410.20779v1
• Date: Mon, 28 Oct 2024 06:40:03 GMT
• Title: Decoding Reading Goals from Eye Movements
• Authors: Omer Shubi, Cfir Avraham Hadar, Yevgeni Berzak,
• Abstract summary: We examine whether it is possible to decode two types of reading goals that are common in daily life: information seeking and ordinary reading. Using large scale eye-tracking data, we apply to this task a wide range of state-of-the-art models for eye movements and text. We find that eye movements contain highly valuable signals for this task.
• Score: 1.3176926720381554
• License:
• Abstract: Readers can have different goals with respect to the text they are reading. Can these goals be decoded from the pattern of their eye movements over the text? In this work, we examine for the first time whether it is possible to decode two types of reading goals that are common in daily life: information seeking and ordinary reading. Using large scale eye-tracking data, we apply to this task a wide range of state-of-the-art models for eye movements and text that cover different architectural and data representation strategies, and further introduce a new model ensemble. We systematically evaluate these models at three levels of generalization: new textual item, new participant, and the combination of both. We find that eye movements contain highly valuable signals for this task. We further perform an error analysis which builds on prior empirical findings on differences between ordinary reading and information seeking and leverages rich textual annotations. This analysis reveals key properties of textual items and participant eye movements that contribute to the difficulty of the task.

Related papers

• Fine-Grained Prediction of Reading Comprehension from Eye Movements [1.2062053320259833]
  We focus on a fine-grained task of predicting reading comprehension from eye movements at the level of a single question over a passage. We tackle this task using three new multimodal language models, as well as a battery of prior models from the literature. The evaluations suggest that although the task is highly challenging, eye movements contain useful signals for fine-grained prediction of reading comprehension.
  arXiv  Detail & Related papers  (2024-10-06T13:55:06Z)
• Visually Grounded Language Learning: a review of language games, datasets, tasks, and models [60.2604624857992]
  Many Vision+Language (V+L) tasks have been defined with the aim of creating models that can ground symbols in the visual modality. In this work, we provide a systematic literature review of several tasks and models proposed in the V+L field.
  arXiv  Detail & Related papers  (2023-12-05T02:17:29Z)
• Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models [56.257840490146]
  ConCue is a novel approach for improving visual feature extraction in HOI detection. We develop a transformer-based feature extraction module with a multi-tower architecture that integrates contextual cues into both instance and interaction detectors.
  arXiv  Detail & Related papers  (2023-11-26T09:11:32Z)
• An Analysis of Reader Engagement in Literary Fiction through Eye Tracking and Linguistic Features [11.805980147608178]
  We analyzed the significance of various qualities of the text in predicting how engaging a reader is likely to find it. Furthering our understanding of what captivates readers in fiction will help better inform models used in creative narrative generation.
  arXiv  Detail & Related papers  (2023-06-06T22:14:59Z)
• Decoding Attention from Gaze: A Benchmark Dataset and End-to-End Models [6.642042615005632]
  Eye-tracking has potential to provide rich behavioral data about human cognition in ecologically valid environments. This paper studies using computer vision tools for "attention decoding", the task of assessing the locus of a participant's overt visual attention over time.
  arXiv  Detail & Related papers  (2022-11-20T12:24:57Z)
• Vision-Language Pre-Training for Boosting Scene Text Detectors [57.08046351495244]
  We specifically adapt vision-language joint learning for scene text detection. We propose to learn contextualized, joint representations through vision-language pre-training. The pre-trained model is able to produce more informative representations with richer semantics.
  arXiv  Detail & Related papers  (2022-04-29T03:53:54Z)
• Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfer [61.34424171458634]
  We study whether integrating visual knowledge into a language model can fill the gap. Our experiments show that visual knowledge transfer can improve performance in both low-resource and fully supervised settings.
  arXiv  Detail & Related papers  (2022-03-14T22:02:40Z)
• CLRGaze: Contrastive Learning of Representations for Eye Movement Signals [0.0]
  We learn feature vectors of eye movements in a self-supervised manner. We adopt a contrastive learning approach and propose a set of data transformations that encourage a deep neural network to discern salient and granular gaze patterns.
  arXiv  Detail & Related papers  (2020-10-25T06:12:06Z)
• COBE: Contextualized Object Embeddings from Narrated Instructional Video [52.73710465010274]
  We propose a new framework for learning Contextualized OBject Embeddings from automatically-transcribed narrations of instructional videos. We leverage the semantic and compositional structure of language by training a visual detector to predict a contextualized word embedding of the object and its associated narration. Our experiments show that our detector learns to predict a rich variety of contextual object information, and that it is highly effective in the settings of few-shot and zero-shot learning.
  arXiv  Detail & Related papers  (2020-07-14T19:04:08Z)
• A Novel Attention-based Aggregation Function to Combine Vision and Language [55.7633883960205]
  We propose a novel fully-attentive reduction method for vision and language. Specifically, our approach computes a set of scores for each element of each modality employing a novel variant of cross-attention. We test our approach on image-text matching and visual question answering, building fair comparisons with other reduction choices.
  arXiv  Detail & Related papers  (2020-04-27T18:09:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.