Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal

• URL: http://arxiv.org/abs/2606.05346v1
• Date: Wed, 03 Jun 2026 18:44:41 GMT
• Title: Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal
• Abstract summary: We show that word-level prediction error (surprisal) and sensitivity to the local momentum of the unfolding interpretation (trajectory extrapolation error) are dissociable components of processing cost.<n>The effect is especially pronounced in garden-path sentences.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human language comprehension unfolds sequentially: each word is processed in the context of those that came before, and the interpretation builds incrementally over time. Surprisal, the negative log probability of a word given its context, has been the dominant predictor of incremental processing cost. But surprisal reduces rich sequential representations to a single scalar at each word, discarding information about the direction in which the interpretation has been evolving. Dynamical-systems approaches suggest that the trajectory of the evolving interpretive state, not just its position at each moment,should shape processing, and language itself may have local momentum, since speakers plan utterances a few words at a time. We introduce trajectory extrapolation error: at each word, we fit a linear trajectory to the preceding hidden states of a transformer language model and measure deviation from the extrapolated path. On the Natural Stories corpus, this measure is nearly orthogonal to surprisal (r = .044) and independently predicts self-paced reading times. The effect is especially pronounced in garden-path sentences, strengthens with model scale (GPT-2 Small to Large), and replicates across architectures with different positional encoding schemes (GPT-2 vs. Pythia/RoPE). A displacement control shows the effect is not reducible to representational change magnitude: displacement and extrapolation error predict in opposite directions. These findings reveal two dissociable components of processing cost: word-level prediction error (surprisal) and sensitivity to the local momentum of the unfolding interpretation (trajectory extrapolation error).

Related papers

• The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priors [11.541745717034667]
  We show that a language model predicts when it has few clues by using a single direction of the unembedding matrix.<n>Projecting the final prediction state onto this direction yields a per-token emphprior loading factor $$.<n>We show that the direction of ignorance is causally active: raising or lowering $$ at the final prediction state steers the prediction toward or away from the unigram prior in KL divergence.
  arXiv  Detail & Related papers  (2026-09-02T05:05:50Z)
• Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis [14.478855009044807]
  Surprisal theory posits that the processing difficulty of a word is determined by its predictability in context.<n> LMs successfully predict reading times in naturalistic text, but systematically underpredict the magnitude of difficulty observed in controlled studies of syntactic ambiguity.<n>This suggests that differences in the number of simultaneous parses available to LMs and humans cannot reconcile LM-based surprisal with human sentence processing.
  arXiv  Detail & Related papers  (2026-05-14T21:44:26Z)
• Probing for Reading Times [60.30314214800421]
  We probe language model representations for human reading times in English, Greek, Hebrew, Russian, and Turkish.<n>We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration.
  arXiv  Detail & Related papers  (2026-04-20T18:12:59Z)
• Revisiting Real-Time Digging-In Effects: No Evidence from NP/Z Garden-Paths [10.217186322563318]
  We report experiments on English NP/Z garden-path sentences using Maze and self-paced reading.<n>We find no evidence for real-time digging-in effects.
  arXiv  Detail & Related papers  (2026-03-24T18:07:46Z)
• On the Role of Context in Reading Time Prediction [50.87306355705826]
  We present a new perspective on how readers integrate context during real-time language comprehension.<n>Our proposals build on surprisal theory, which posits that the processing effort of a linguistic unit is an affine function of its in-context information content.
  arXiv  Detail & Related papers  (2024-09-12T15:52:22Z)
• Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language [2.1756081703276]
  We test a hypothesis about predictive representations of autoregressive transformers. Key insight is that straighter trajectories should facilitate prediction via linear extrapolation. We quantify straightness using a 1-dimensional curvature metric.
  arXiv  Detail & Related papers  (2023-11-05T22:16:21Z)
• Humans and language models diverge when predicting repeating text [52.03471802608112]
  We present a scenario in which the performance of humans and LMs diverges. Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory begins to play a role. We hope that this scenario will spur future work in bringing LMs closer to human behavior.
  arXiv  Detail & Related papers  (2023-10-10T08:24:28Z)
• Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? [9.909170013118775]
  The propensity of larger Transformer-based models to'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations. These results suggest that the propensity of larger Transformer-based models to'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations.
  arXiv  Detail & Related papers  (2022-12-23T03:57:54Z)
• Neuro-Symbolic Entropy Regularization [78.16196949641079]
  In structured prediction, the goal is to jointly predict many output variables that together encode a structured object. One approach -- entropy regularization -- posits that decision boundaries should lie in low-probability regions. We propose a loss, neuro-symbolic entropy regularization, that encourages the model to confidently predict a valid object.
  arXiv  Detail & Related papers  (2022-01-25T06:23:10Z)
• Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing [18.91129968022831]
  Interpretability methods need to be robust for trustworthy NLP applications in high-stake areas like medicine or finance. Our paper demonstrates how interpretations can be manipulated by making simple word perturbations on an input text.
  arXiv  Detail & Related papers  (2021-08-11T02:07:21Z)
• Constructing interval variables via faceted Rasch measurement and multitask deep learning: a hate speech application [63.10266319378212]
  We propose a method for measuring complex variables on a continuous, interval spectrum by combining supervised deep learning with the Constructing Measures approach to faceted Rasch item response theory (IRT) We demonstrate this new method on a dataset of 50,000 social media comments sourced from YouTube, Twitter, and Reddit and labeled by 11,000 U.S.-based Amazon Mechanical Turk workers.
  arXiv  Detail & Related papers  (2020-09-22T02:15:05Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.