Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis

• URL: http://arxiv.org/abs/2605.15440v1
• Date: Thu, 14 May 2026 21:44:26 GMT
• Title: Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis
• Abstract summary: Surprisal theory posits that the processing difficulty of a word is determined by its predictability in context.<n> LMs successfully predict reading times in naturalistic text, but systematically underpredict the magnitude of difficulty observed in controlled studies of syntactic ambiguity.<n>This suggests that differences in the number of simultaneous parses available to LMs and humans cannot reconcile LM-based surprisal with human sentence processing.
• Score: 14.478855009044807
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Surprisal theory posits that the processing difficulty of a word is determined by its predictability in context, offering a potential link between human sentence processing and next-word predictions from language models. While language model (LM) surprisals successfully predict reading times in naturalistic text, they systematically underpredict the magnitude of difficulty observed in controlled studies of syntactic ambiguity, particularly in garden path sentences. This mismatch might arise from differences in the computational constraints between humans and LMs. Here we test one such hypothesis, specifically, that LMs may be able to simultaneously consider a greater number of distinct sentence interpretations at once, compared to humans. Using Recurrent Neural Network Grammars (RNNGs) with word-synchronous beam search, we systematically vary the number of simultaneous parses used to compute word surprisal, and then use these surprisals to predict human reading times. Reducing the number of simultaneous active parses indeed increases the magnitude of predicted garden path effects, but not nearly enough to capture the full magnitude of the effects in humans. This suggests that differences in the number of simultaneous parses available to LMs and humans cannot reconcile LM-based surprisal with human sentence processing.

Related papers

• Probing for Reading Times [60.30314214800421]
  We probe language model representations for human reading times in English, Greek, Hebrew, Russian, and Turkish.<n>We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration.
  arXiv  Detail & Related papers  (2026-04-20T18:12:59Z)
• An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal [21.29543663048103]
  We investigate whether it is impossible to construct a neural LM that can explain garden-path effects via surprisal.<n>Our results show that fine-tuned LMs do not overfit and successfully capture human reading slowdowns on held-out garden-path items.
  arXiv  Detail & Related papers  (2026-04-20T14:01:11Z)
• Predicting Sentence Acceptability Judgments in Multimodal Contexts [22.053970196200925]
  Previous work has examined the capacity of deep neural networks (DNNs) to predict human sentence acceptability judgments.<n>We consider the effect of prior exposure to visual images (i.e., visual context) on these judgments for humans and large language models (LLMs)<n>Our results suggest that, in contrast to textual context, visual images appear to have little if any impact on human acceptability ratings.
  arXiv  Detail & Related papers  (2026-02-24T13:54:38Z)
• On the Role of Context in Reading Time Prediction [50.87306355705826]
  We present a new perspective on how readers integrate context during real-time language comprehension.<n>Our proposals build on surprisal theory, which posits that the processing effort of a linguistic unit is an affine function of its in-context information content.
  arXiv  Detail & Related papers  (2024-09-12T15:52:22Z)
• Multipath parsing in the brain [4.605070569473395]
  Humans understand sentences word-by-word, in the order that they hear them. We investigate how humans process these syntactic ambiguities by correlating predictions from incremental dependencys with timecourse data from people undergoing functional neuroimaging while listening to an audiobook. In both English and Chinese, we find evidence for multipath parsing. Brain regions associated with this multipath effect include bilateral superior temporal gyrus.
  arXiv  Detail & Related papers  (2024-01-31T18:07:12Z)
• Humans and language models diverge when predicting repeating text [52.03471802608112]
  We present a scenario in which the performance of humans and LMs diverges. Human and GPT-2 LM predictions are strongly aligned in the first presentation of a text span, but their performance quickly diverges when memory begins to play a role. We hope that this scenario will spur future work in bringing LMs closer to human behavior.
  arXiv  Detail & Related papers  (2023-10-10T08:24:28Z)
• Testing the Predictions of Surprisal Theory in 11 Languages [77.45204595614]
  We investigate the relationship between surprisal and reading times in eleven different languages.<n>By focusing on a more diverse set of languages, we argue that these results offer the most robust link to-date between information theory and incremental language processing across languages.
  arXiv  Detail & Related papers  (2023-07-07T15:37:50Z)
• Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? [9.909170013118775]
  The propensity of larger Transformer-based models to'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations. These results suggest that the propensity of larger Transformer-based models to'memorize' sequences during training makes their surprisal estimates diverge from humanlike expectations.
  arXiv  Detail & Related papers  (2022-12-23T03:57:54Z)
• Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities [19.659811811023374]
  We propose a method for estimating syntactic predictability from a language model. We find that treating syntactic predictability independently from lexical predictability indeed results in larger estimates of garden path effects. Our results support the hypothesis that predictability is not the only factor responsible for the processing cost associated with garden path sentences.
  arXiv  Detail & Related papers  (2022-10-21T18:30:56Z)
• Evaluating Distributional Distortion in Neural Language Modeling [81.83408583979745]
  A heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language. Standard language modeling metrics such as perplexity quantify the performance of language models (LM) in aggregate. We develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages.
  arXiv  Detail & Related papers  (2022-03-24T01:09:46Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.