The Frequency Confound in Language-Model Surprisal and Metaphor Novelty

• URL: http://arxiv.org/abs/2605.06506v1
• Date: Thu, 07 May 2026 16:20:37 GMT
• Title: The Frequency Confound in Language-Model Surprisal and Metaphor Novelty
• Abstract summary: We analyse surprisal estimates from eight Pythia model sizes and 154 training checkpoints.<n>Across settings, word frequency is a stronger predictor of metaphor novelty than surprisal.<n>These results suggest that the often-reported optimal LM surprisal settings may incorrectly associate contextual predictability with metaphor novelty and processing difficulty, whereas lexical frequency may be the major underlying factor.
• Score: 12.10361869131849
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language-model (LM) surprisal is widely used as a proxy for contextual predictability and has been reported to correlate with metaphor novelty judgments. However, surprisal is tightly intertwined with lexical frequency. We explore this interaction on metaphor novelty ratings using two different word frequency measures. We analyse surprisal estimates from eight Pythia model sizes and 154 training checkpoints. Across settings, word frequency is a stronger predictor of metaphor novelty than surprisal. Across training stages, the surprisal--novelty association peaks at an early stage and then falls again, mirroring a similarly timed increase in the surprisal--frequency association. These results suggest that the often-reported optimal LM surprisal settings may incorrectly associate contextual predictability with metaphor novelty and processing difficulty, whereas lexical frequency may be the major underlying factor.

Related papers

• From Exposure to Expectation: Frequency, Surprisal, and Language Across Development in Spanish [0.0]
  Surprisal, the negative log-probability a language model assigns to a word given its preceding context, reliably predicts adult reading times.<n>We investigate this question across two corpus-based studies of Spanish.
  arXiv  Detail & Related papers  (2026-08-23T15:02:29Z)
• Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis [14.478855009044807]
  Surprisal theory posits that the processing difficulty of a word is determined by its predictability in context.<n> LMs successfully predict reading times in naturalistic text, but systematically underpredict the magnitude of difficulty observed in controlled studies of syntactic ambiguity.<n>This suggests that differences in the number of simultaneous parses available to LMs and humans cannot reconcile LM-based surprisal with human sentence processing.
  arXiv  Detail & Related papers  (2026-05-14T21:44:26Z)
• Probing for Reading Times [60.30314214800421]
  We probe language model representations for human reading times in English, Greek, Hebrew, Russian, and Turkish.<n>We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration.
  arXiv  Detail & Related papers  (2026-04-20T18:12:59Z)
• Surprisal and Metaphor Novelty: Moderate Correlations and Divergent Scaling Effects [9.85813493536013]
  This study investigates whether surprisal, a probabilistic measure of predictability in language models, correlates with metaphor novelty datasets.<n>We analyse surprisal from 16 LM variants on corpus-based and synthetic metaphor novelty datasets.
  arXiv  Detail & Related papers  (2026-01-05T11:24:33Z)
• On the Proper Treatment of Tokenization in Psycholinguistics [53.960910019072436]
  The paper argues that token-level language models should be marginalized into character-level language models before they are used in psycholinguistic studies.<n>We find various focal areas whose surprisal is a better psychometric predictor than the surprisal of the region of interest itself.
  arXiv  Detail & Related papers  (2024-10-03T17:18:03Z)
• On the Role of Context in Reading Time Prediction [50.87306355705826]
  We present a new perspective on how readers integrate context during real-time language comprehension.<n>Our proposals build on surprisal theory, which posits that the processing effort of a linguistic unit is an affine function of its in-context information content.
  arXiv  Detail & Related papers  (2024-09-12T15:52:22Z)
• Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times [15.738530737312335]
  Recent studies have shown that as Transformer-based language models become larger and are trained on very large amounts of data, the fit of their surprisal estimates to naturalistic human reading times degrades. This paper presents a series of analyses showing that word frequency is a key explanatory factor underlying these two trends. The results indicate that Transformer-based language models' surprisal estimates diverge from human-like expectations due to the superhumanly complex associations they learn for predicting rare words.
  arXiv  Detail & Related papers  (2024-02-03T20:22:54Z)
• Testing the Predictions of Surprisal Theory in 11 Languages [77.45204595614]
  We investigate the relationship between surprisal and reading times in eleven different languages.<n>By focusing on a more diverse set of languages, we argue that these results offer the most robust link to-date between information theory and incremental language processing across languages.
  arXiv  Detail & Related papers  (2023-07-07T15:37:50Z)
• Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis [69.07674653828565]
  Machine learning models have a tendency to leverage spurious correlations that exist in the training set but may not hold true in general circumstances. In this paper, we examine the implications of spurious correlations through a novel perspective called neighborhood analysis. We propose a family of regularization methods, NFL (doN't Forget your Language) to mitigate spurious correlations in text classification.
  arXiv  Detail & Related papers  (2023-05-23T03:55:50Z)
• On Sampling-Based Training Criteria for Neural Language Modeling [97.35284042981675]
  We consider Monte Carlo sampling, importance sampling, a novel method we call compensated partial summation, and noise contrastive estimation. We show that all these sampling methods can perform equally well, as long as we correct for the intended class posterior probabilities. Experimental results in language modeling and automatic speech recognition on Switchboard and LibriSpeech support our claim.
  arXiv  Detail & Related papers  (2021-04-21T12:55:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.