Modeling Programmer Attention as Scanpath Prediction

• URL: http://arxiv.org/abs/2308.13920v1
• Date: Sat, 26 Aug 2023 17:35:19 GMT
• Title: Modeling Programmer Attention as Scanpath Prediction
• Abstract summary: This paper launches a new effort at modeling programmer attention by predicting eye movement scanpaths. Models of programmer attention are important because they help researchers build better interfaces, assistive technologies, and more human-like AI. We present an eye tracking experiment with 27 programmers and a prototype scanpath predictor to present preliminary results and obtain early community feedback.
• Score: 14.81050529571135
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper launches a new effort at modeling programmer attention by predicting eye movement scanpaths. Programmer attention refers to what information people intake when performing programming tasks. Models of programmer attention refer to machine prediction of what information is important to people. Models of programmer attention are important because they help researchers build better interfaces, assistive technologies, and more human-like AI. For many years, researchers in SE have built these models based on features such as mouse clicks, key logging, and IDE interactions. Yet the holy grail in this area is scanpath prediction -- the prediction of the sequence of eye fixations a person would take over a visual stimulus. A person's eye movements are considered the most concrete evidence that a person is taking in a piece of information. Scanpath prediction is a notoriously difficult problem, but we believe that the emergence of lower-cost, higher-accuracy eye tracking equipment and better large language models of source code brings a solution within grasp. We present an eye tracking experiment with 27 programmers and a prototype scanpath predictor to present preliminary results and obtain early community feedback.

Related papers

• Predicting Human Visual Attention on Words in Source Code [0.7448254811651417]
  We design a novel loss function that computes similarity between human attention measured during eye tracking experiments and the internal attention of the artificial neural network.<n>Our model achieves a statistically significant improvement over the close baseline in the reading task according to normalized Levenshtein distance.
  arXiv  Detail & Related papers  (2026-07-16T03:42:16Z)
• Modeling Human Gaze Behavior with Diffusion Models for Unified Scanpath Prediction [66.71402249062777]
  We present ScanDiff, a novel architecture that combines diffusion models with Vision Transformers to generate diverse and realistic scanpaths.<n>Our method explicitly models scanpath variability by leveraging the nature of diffusion models, producing a wide range of plausible gaze trajectories.<n>Experiments on benchmark datasets show that ScanDiff surpasses state-of-the-art methods in both free-viewing and task-driven scenarios.
  arXiv  Detail & Related papers  (2025-07-30T18:36:09Z)
• Human Scanpath Prediction in Target-Present Visual Search with Semantic-Foveal Bayesian Attention [49.99728312519117]
  SemBA-FAST is a top-down framework designed for predicting human visual attention in target-present visual search.<n>We evaluate SemBA-FAST on the COCO-Search18 benchmark dataset, comparing its performance against other scanpath prediction models.<n>These findings provide valuable insights into the capabilities of semantic-foveal probabilistic frameworks for human-like attention modelling.
  arXiv  Detail & Related papers  (2025-07-24T15:19:23Z)
• DiSciPLE: Learning Interpretable Programs for Scientific Visual Discovery [61.02102713094486]
  Good interpretation is important in scientific reasoning, as it allows for better decision-making. This paper introduces an automatic way of obtaining such interpretable-by-design models, by learning programs that interleave neural networks. We propose DiSciPLE an evolutionary algorithm that leverages common sense and prior knowledge of large language models (LLMs) to create Python programs explaining visual data.
  arXiv  Detail & Related papers  (2025-02-14T10:26:14Z)
• Look Hear: Gaze Prediction for Speech-directed Human Attention [49.81718760025951]
  Our study focuses on the incremental prediction of attention as a person is seeing an image and hearing a referring expression. We developed the Attention in Referral Transformer model or ART, which predicts the human fixations spurred by each word in a referring expression. In our quantitative and qualitative analyses, ART not only outperforms existing methods in scanpath prediction, but also appears to capture several human attention patterns.
  arXiv  Detail & Related papers  (2024-07-28T22:35:08Z)
• OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction [0.2796197251957245]
  This paper introduces the Object-level Attention Transformer (OAT) OAT predicts human scanpaths as they search for a target object within a cluttered scene of distractors. We evaluate OAT on the Amazon book cover dataset and a new dataset for visual search that we collected.
  arXiv  Detail & Related papers  (2024-07-18T09:33:17Z)
• EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning [31.583764158565916]
  We present EyeFormer, a machine learning model for predicting scanpaths in a visual user interface. Our model has the unique capability of producing personalized predictions when given a few user scanpath samples. It can predict full scanpath information, including fixation positions and duration, across individuals and various stimulus types.
  arXiv  Detail & Related papers  (2024-04-15T22:26:27Z)
• Social-Transmotion: Promptable Human Trajectory Prediction [65.80068316170613]
  Social-Transmotion is a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior. Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY.
  arXiv  Detail & Related papers  (2023-12-26T18:56:49Z)
• Enhancing Robot Learning through Learned Human-Attention Feature Maps [6.724036710994883]
  We think that embedding auxiliary information about focus point into robot learning would enhance efficiency and robustness of the learning process. In this paper, we propose a novel approach to model and emulate the human attention with an approximate prediction model. We test our approach on two learning tasks - object detection and imitation learning.
  arXiv  Detail & Related papers  (2023-08-29T14:23:44Z)
• ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understanding [67.21613160846299]
  Embodied Reference Understanding (ERU) is first designed for this concern. New dataset called ScanERU is constructed to evaluate the effectiveness of this idea.
  arXiv  Detail & Related papers  (2023-03-23T11:36:14Z)
• Target-absent Human Attention [44.10971508325032]
  We propose the first data-driven computational model that addresses the search-termination problem. We represent the internal knowledge that the viewer acquires through fixations using a novel state representation. We improve the state of the art in predicting human target-absent search behavior on the COCO-Search18 dataset.
  arXiv  Detail & Related papers  (2022-07-04T02:32:04Z)
• Pedestrian 3D Bounding Box Prediction [83.7135926821794]
  We focus on 3D bounding boxes, which are reasonable estimates of humans without modeling complex motion details for autonomous vehicles. We suggest this new problem and present a simple yet effective model for pedestrians' 3D bounding box prediction. This method follows an encoder-decoder architecture based on recurrent neural networks.
  arXiv  Detail & Related papers  (2022-06-28T17:59:45Z)
• Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? [54.442692221567796]
  Task specification is critical for engagement of non-expert end-users and adoption of personalized robots. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use.
  arXiv  Detail & Related papers  (2022-04-23T19:39:49Z)
• From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video [4.163207534602723]
  Deciphering human behaviors to predict their future paths/trajectories is important. Human trajectory prediction still remains a challenging task. System must be able to detect and analyze human activities as well as scene semantics.
  arXiv  Detail & Related papers  (2020-11-20T22:23:34Z)
• Projection Mapping Implementation: Enabling Direct Externalization of Perception Results and Action Intent to Improve Robot Explainability [62.03014078810652]
  Existing research on non-verbal cues, e.g., eye gaze or arm movement, may not accurately present a robot's internal states. Projecting the states directly onto a robot's operating environment has the advantages of being direct, accurate, and more salient.
  arXiv  Detail & Related papers  (2020-10-05T18:16:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.