Probing for Reading Times

• URL: http://arxiv.org/abs/2604.18712v1
• Date: Mon, 20 Apr 2026 18:12:59 GMT
• Title: Probing for Reading Times
• Abstract summary: We probe language model representations for human reading times in English, Greek, Hebrew, Russian, and Turkish.<n>We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration.
• Score: 60.30314214800421
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Probing has shown that language model representations encode rich linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. In this work, we probe language model representations for human reading times. Using regularized linear regression on two eye-tracking corpora spanning five languages (English, Greek, Hebrew, Russian, and Turkish), we compare the representations from every model layer against scalar predictors -- surprisal, information value, and logit-lens surprisal. We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. The concentration of predictive power in the early layers suggests that human-like processing signatures are captured by low-level structural or lexical representations, pointing to a functional alignment between model depth and the temporal stages of human reading. In contrast, for late-pass measures such as total reading time, scalar surprisal remains superior, despite its being a much more compressed representation. We also observe performance gains when using both surprisal and early-layer representations. Overall, we find that the best-performing predictor varies strongly depending on the language and eye-tracking measure.

Related papers

• The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage [13.411790077470222]
  It has been speculated that data leakage may be due to data leakage that caused language models to see the text stimuli during training.<n>This paper presents two studies to address this concern at scale.<n>The first study reveals relatively little leakage of five naturalistic reading time corpora in two pre-training datasets in terms of length and frequency of token $n$-gram overlap.<n>The second study replicates the negative relationship between language model size and the fit of surprisal to reading times using models trained on 'leakage-free' data that overlaps only minimally with the reading time corpora.
  arXiv  Detail & Related papers  (2025-06-01T21:12:36Z)
• Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"? [15.773775387121097]
  We show that calibration of large language models typically improves with model size. We find that temperature-scaling probabilities lead to a systematically better fit to reading times.
  arXiv  Detail & Related papers  (2023-11-15T19:34:06Z)
• Humans and language models diverge when predicting repeating text [52.03471802608112]
  We present a scenario in which the performance of humans and LMs diverges. Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory begins to play a role. We hope that this scenario will spur future work in bringing LMs closer to human behavior.
  arXiv  Detail & Related papers  (2023-10-10T08:24:28Z)
• Testing the Predictions of Surprisal Theory in 11 Languages [77.45204595614]
  We investigate the relationship between surprisal and reading times in eleven different languages.<n>By focusing on a more diverse set of languages, we argue that these results offer the most robust link to-date between information theory and incremental language processing across languages.
  arXiv  Detail & Related papers  (2023-07-07T15:37:50Z)
• Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? [9.909170013118775]
  The propensity of larger Transformer-based models to'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations. These results suggest that the propensity of larger Transformer-based models to'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations.
  arXiv  Detail & Related papers  (2022-12-23T03:57:54Z)
• A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis [90.24921443175514]
  We focus on aspect-based sentiment analysis, which involves extracting aspect term, category, and predicting their corresponding polarities. We propose to reformulate the extraction and prediction tasks into the sequence generation task, using a generative language model with unidirectional attention. Our approach outperforms the previous state-of-the-art (based on BERT) on average performance by a large margins in few-shot and full-shot settings.
  arXiv  Detail & Related papers  (2022-04-11T18:31:53Z)
• Multilingual Language Models Predict Human Reading Behavior [8.830621849672108]
  We compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures. We find that BERT and XLM models successfully predict a range of eye tracking features. In a series of experiments, we analyze the cross-domain and cross-language abilities of these models and show how they reflect human sentence processing.
  arXiv  Detail & Related papers  (2021-04-12T13:03:49Z)
• Multi-timescale Representation Learning in LSTM Language Models [69.98840820213937]
  Language models must capture statistical dependencies between words at timescales ranging from very short to very long. We derived a theory for how the memory gating mechanism in long short-term memory language models can capture power law decay. Experiments showed that LSTM language models trained on natural English text learn to approximate this theoretical distribution.
  arXiv  Detail & Related papers  (2020-09-27T02:13:38Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.