Surprisal Theory is Tautological (without Rational Grounding)

• URL: http://arxiv.org/abs/2607.21574v1
• Date: Thu, 23 Jul 2026 17:54:37 GMT
• Title: Surprisal Theory is Tautological (without Rational Grounding)
• Abstract summary: Surprisal theory holds that the human processing difficulty of a linguistic unit in context is an affine function of its surprisal under some language model.<n>I argue this claim is a tautology without further constraint.<n>I conclude that breaking the tautology requires a rationalist intervention.
• Score: 60.917028769172795
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Surprisal theory holds that the human processing difficulty of a linguistic unit in context is an affine function of its surprisal under some language model. I argue this claim is a tautology without further constraint: for any non-negative difficulty measure over units in context, there exists a language model whose surprisal is an affine function of it under mild technical conditions. Therefore, because any pattern of difficulty is consistent with some language model, without an additional constraint on the language model, surprisal theory makes no falsifiable predictions. The tautology was long obscured by an assumption implicit in two decades of psycholinguistic work---that the relevant language model is the distribution that generated the training corpus, so that improving corpus fit improves predictions of human behavior. Recent empirical work has undermined this assumption, demonstrating that better corpus models can be worse predictors of processing difficulty. I conclude that breaking the tautology requires a rationalist intervention, i.e., the relevant language model must be derived from a non-empirically motivated model of the comprehender, which could be based on, for instance, memory constraints or processing goals, and that, thus, does not depend on the behavioral data surprisal theory is meant to explain.

Related papers

• Learning inflection classes using Adaptive Resonance Theory [0.2676349883103403]
  We study the learnability of a system of verbal inflection classes by the individual language user.<n>We use Adaptive Resonance Theory, a neural network with a parameter that determines the degree of generalisation (vigilance)<n>The similarity of clustering to attested inflection classes varies depending on the complexity of the inflectional system.
  arXiv  Detail & Related papers  (2025-12-17T15:58:20Z)
• From Black-box to Causal-box: Towards Building More Interpretable Models [57.23201263629627]
  We introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models.<n>We derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query.
  arXiv  Detail & Related papers  (2025-10-24T20:03:18Z)
• Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning [0.0]
  We study logical reasoning in language models by asking whether their errors follow established human fallacy.<n>For each response, we judge logical reasoning and correctness when it matches an ETRpredicted fallacy.
  arXiv  Detail & Related papers  (2025-06-10T17:04:33Z)
• Counterfactual reasoning: an analysis of in-context emergence [57.118735341305786]
  We show that language models are capable of counterfactual reasoning.<n>We find that self-attention, model depth and pre-training data diversity drive performance.<n>Our findings extend to counterfactual reasoning under SDE dynamics.
  arXiv  Detail & Related papers  (2025-06-05T16:02:07Z)
• Perceptions of Linguistic Uncertainty by Language Models and Humans [26.69714008538173]
  We investigate how language models map linguistic expressions of uncertainty to numerical responses. We find that 7 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge.
  arXiv  Detail & Related papers  (2024-07-22T17:26:12Z)
• UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations [62.71847873326847]
  We investigate the ability to model unusual, unexpected, and unlikely situations. Given a piece of context with an unexpected outcome, this task requires reasoning abductively to generate an explanation. We release a new English language corpus called UNcommonsense.
  arXiv  Detail & Related papers  (2023-11-14T19:00:55Z)
• Testing the Predictions of Surprisal Theory in 11 Languages [77.45204595614]
  We investigate the relationship between surprisal and reading times in eleven different languages.<n>By focusing on a more diverse set of languages, we argue that these results offer the most robust link to-date between information theory and incremental language processing across languages.
  arXiv  Detail & Related papers  (2023-07-07T15:37:50Z)
• State space models can express n-gram languages [51.823427608117626]
  We build state space language models that can solve the next-word prediction task for languages generated from n-gram rules.<n>Our proof shows how SSMs can encode n-gram rules using new theoretical results on their capacity.<n>We conduct experiments with a small dataset generated from n-gram rules to show how our framework can be applied to SSMs and RNNs obtained through gradient-based optimization.
  arXiv  Detail & Related papers  (2023-06-20T10:41:23Z)
• Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning [9.391375268580806]
  We show that competing linguistic processes within a language obscure underlying linguistic knowledge. While human behavior has been found to be similar across languages, we find cross-linguistic variation in model behavior. Our results suggest that models need to learn both the linguistic constraints in a language and their relative ranking, with mismatches in either producing non-human-like behavior.
  arXiv  Detail & Related papers  (2021-06-02T14:52:11Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)
• CausaLM: Causal Model Explanation Through Counterfactual Language Models [33.29636213961804]
  CausaLM is a framework for producing causal model explanations using counterfactual language representation models. We show that language representation models such as BERT can effectively learn a counterfactual representation for a given concept of interest. A byproduct of our method is a language representation model that is unaffected by the tested concept.
  arXiv  Detail & Related papers  (2020-05-27T15:06:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.