Predicting Human Visual Attention on Words in Source Code

• URL: http://arxiv.org/abs/2607.14535v1
• Date: Thu, 16 Jul 2026 03:42:16 GMT
• Title: Predicting Human Visual Attention on Words in Source Code
• Abstract summary: We design a novel loss function that computes similarity between human attention measured during eye tracking experiments and the internal attention of the artificial neural network.<n>Our model achieves a statistically significant improvement over the close baseline in the reading task according to normalized Levenshtein distance.
• Score: 0.7448254811651417
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a computational model to predict human visual attention over words in software source code. The visual attention of software engineers when reading source code has long been studied as a means to understand human cognitive processes during software engineering tasks. Predicting this visual attention is important for perfecting user interface design and understanding what information human programmers need. We propose a model of programmer visual attention in which we design a novel loss function that computes similarity between human attention measured during eye tracking experiments and the internal attention of the artificial neural network. We evaluate our model by comparing its outputs to actual eye tracking data from three separate datasets. Two are in the Java programming language and one is in the C programming language. Our model outperforms the baseline in software engineering by 64%, 16%, and 467% in each of these studies according to Pearson correlation. We used scanpath prediction as an example to demonstrate that our model is more capable of the task that requires the understanding of human thought process. Our model achieves a statistically significant improvement over the close baseline in the reading task according to normalized Levenshtein distance and outperforms both Claude and GPT-5 on both reading and writing tasks.

Related papers

• Human-level 3D shape perception emerges from multi-view learning [63.048728487674815]
  We develop a modeling framework that predicts human 3D shape inferences for arbitrary objects.<n>We achieve this with a novel class of neural networks trained using a visual-spatial objective over naturalistic sensory data.<n>We find that human-level 3D perception can emerge from a simple, scalable learning objective over naturalistic visual-spatial data.
  arXiv  Detail & Related papers  (2026-02-19T18:56:05Z)
• EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention [27.11897727181663]
  We present EyeMulator, a technique for training CodeLLMs to mimic human visual attention while training for various software development tasks.<n>We draw these weights from observations of human visual attention derived from a previously-collected publicly-available dataset of eye-tracking experiments in software engineering tasks.<n>Our evaluation shows that EyeMulator outperforms strong LLM baselines on several tasks such as code translation, completion and summarization.
  arXiv  Detail & Related papers  (2025-08-22T20:08:09Z)
• Human Scanpath Prediction in Target-Present Visual Search with Semantic-Foveal Bayesian Attention [49.99728312519117]
  SemBA-FAST is a top-down framework designed for predicting human visual attention in target-present visual search.<n>We evaluate SemBA-FAST on the COCO-Search18 benchmark dataset, comparing its performance against other scanpath prediction models.<n>These findings provide valuable insights into the capabilities of semantic-foveal probabilistic frameworks for human-like attention modelling.
  arXiv  Detail & Related papers  (2025-07-24T15:19:23Z)
• CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models [18.891323067948285]
  It is unclear to what degree vision-language models emulate human behavior on tasks that involve reasoning about data visualizations.<n>Here we evaluated eight vision-language models on six data visualization literacy assessments designed for humans.<n>We found that these models performed worse than human participants on average.
  arXiv  Detail & Related papers  (2025-05-22T18:15:04Z)
• Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding [94.64781599202882]
  Vision Language Models (VLMs) have achieved remarkable progress in multimodal tasks.<n>They often struggle with visual arithmetic, seemingly simple capabilities like object counting or length comparison.<n>We propose CogAlign, a novel post-training strategy inspired by Piaget's theory of cognitive development.
  arXiv  Detail & Related papers  (2025-02-17T06:54:49Z)
• DiSciPLE: Learning Interpretable Programs for Scientific Visual Discovery [61.02102713094486]
  Good interpretation is important in scientific reasoning, as it allows for better decision-making.<n>This paper introduces an automatic way of obtaining such interpretable-by-design models, by learning programs that interleave neural networks.<n>We propose DiSciPLE an evolutionary algorithm that leverages common sense and prior knowledge of large language models (LLMs) to create Python programs explaining visual data.
  arXiv  Detail & Related papers  (2025-02-14T10:26:14Z)
• Exploiting Contextual Uncertainty of Visual Data for Efficient Training of Deep Models [0.65268245109828]
  We introduce the notion of contextual diversity for active learning CDAL. We propose a data repair algorithm to curate contextually fair data to reduce model bias. We are working on developing image retrieval system for wildlife camera trap images and reliable warning system for poor quality rural roads.
  arXiv  Detail & Related papers  (2024-11-04T09:43:33Z)
• Visual Grounding Helps Learn Word Meanings in Low-Data Regimes [47.7950860342515]
  Modern neural language models (LMs) are powerful tools for modeling human sentence production and comprehension. But to achieve these results, LMs must be trained in distinctly un-human-like ways. Do models trained more naturalistically -- with grounded supervision -- exhibit more humanlike language learning? We investigate this question in the context of word learning, a key sub-task in language acquisition.
  arXiv  Detail & Related papers  (2023-10-20T03:33:36Z)
• Modeling Programmer Attention as Scanpath Prediction [14.81050529571135]
  This paper launches a new effort at modeling programmer attention by predicting eye movement scanpaths. Models of programmer attention are important because they help researchers build better interfaces, assistive technologies, and more human-like AI. We present an eye tracking experiment with 27 programmers and a prototype scanpath predictor to present preliminary results and obtain early community feedback.
  arXiv  Detail & Related papers  (2023-08-26T17:35:19Z)
• Visual Distant Supervision for Scene Graph Generation [66.10579690929623]
  Scene graph models usually require supervised learning on large quantities of labeled data with intensive human annotation. We propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data. Comprehensive experimental results show that our distantly supervised model outperforms strong weakly supervised and semi-supervised baselines.
  arXiv  Detail & Related papers  (2021-03-29T06:35:24Z)
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions [50.435861435121915]
  We use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. Our experiments show that our "muscly-supervised" representation outperforms a visual-only state-of-the-art method MoCo.
  arXiv  Detail & Related papers  (2020-10-16T17:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.