Surprisal and Metaphor Novelty: Moderate Correlations and Divergent Scaling Effects

• URL: http://arxiv.org/abs/2601.02015v2
• Date: Thu, 08 Jan 2026 18:27:27 GMT
• Title: Surprisal and Metaphor Novelty: Moderate Correlations and Divergent Scaling Effects
• Abstract summary: This study investigates whether surprisal, a probabilistic measure of predictability in language models, correlates with metaphor novelty datasets.<n>We analyse surprisal from 16 LM variants on corpus-based and synthetic metaphor novelty datasets.
• Score: 9.85813493536013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Novel metaphor comprehension involves complex semantic processes and linguistic creativity, making it an interesting task for studying language models (LMs). This study investigates whether surprisal, a probabilistic measure of predictability in LMs, correlates with different metaphor novelty datasets. We analyse surprisal from 16 LM variants on corpus-based and synthetic metaphor novelty datasets. We explore a cloze-style surprisal method that conditions on full-sentence context. Results show that LMs yield significant moderate correlations with scores/labels of metaphor novelty. We further identify divergent scaling patterns: on corpus-based data, correlation strength decreases with model size (inverse scaling effect), whereas on synthetic data it increases (Quality-Power Hypothesis). We conclude that while surprisal can partially account for annotations of metaphor novelty, it remains a limited metric of linguistic creativity.

Related papers

• Modeling semantic association in self-paced reading with language model embeddings [1.647226581907515]
  We use embeddings from LMs to estimate semantic association on a corpus of electroencephalography (EEG) and self-paced reading of natural, Dutch texts.<n>The choice of embedding model can alter the estimated effect of semantic association on both the N400 and self-paced reading times.
  arXiv  Detail & Related papers  (2026-06-05T09:06:22Z)
• The Frequency Confound in Language-Model Surprisal and Metaphor Novelty [12.10361869131849]
  We analyse surprisal estimates from eight Pythia model sizes and 154 training checkpoints.<n>Across settings, word frequency is a stronger predictor of metaphor novelty than surprisal.<n>These results suggest that the often-reported optimal LM surprisal settings may incorrectly associate contextual predictability with metaphor novelty and processing difficulty, whereas lexical frequency may be the major underlying factor.
  arXiv  Detail & Related papers  (2026-05-07T16:20:37Z)
• Floating or Suggesting Ideas? A Large-Scale Contrastive Analysis of Metaphorical and Literal Verb-Object Constructions [53.690096725532726]
  We analyze 297 English verb-object pairs (e.g., float idea vs. suggest idea) in 2M corpus sentences.<n>Using five NLP tools, we extract 2,293 cognitive and linguistic features capturing affective, lexical, syntactic, and discourse-level properties.<n>Cross-pair results show literal contexts have higher lexical frequency, cohesion, and structural regularity, while metaphorical contexts show greater affective load, imageability, lexical diversity, and constructional specificity.
  arXiv  Detail & Related papers  (2026-04-09T14:08:57Z)
• Hybrid topic modelling for computational close reading: Mapping narrative themes in Pushkin's Evgenij Onegin [0.0]
  This study presents a hybrid topic modelling framework for computational literary analysis.<n>It integrates Latent Dirichlet Allocation (LDA) with sparse Partial Least Squares Discriminant Analysis (sPLS-DA) to model thematic structure and longitudinal dynamics in narrative poetry.
  arXiv  Detail & Related papers  (2026-03-20T13:37:08Z)
• Explaining Text Similarity in Transformer Models [52.571158418102584]
  Recent advances in explainable AI have made it possible to mitigate limitations by leveraging improved explanations for Transformers. We use BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, to investigate which feature interactions drive similarity in NLP models. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.
  arXiv  Detail & Related papers  (2024-05-10T17:11:31Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis [69.07674653828565]
  Machine learning models have a tendency to leverage spurious correlations that exist in the training set but may not hold true in general circumstances. In this paper, we examine the implications of spurious correlations through a novel perspective called neighborhood analysis. We propose a family of regularization methods, NFL (doN't Forget your Language) to mitigate spurious correlations in text classification.
  arXiv  Detail & Related papers  (2023-05-23T03:55:50Z)
• Attention-likelihood relationship in transformers [2.8304391396200064]
  We analyze how large language models (LLMs) represent out-of-context words, investigating their reliance on the given context to capture their semantics. Our likelihood-guided text perturbations reveal a correlation between token likelihood and attention values in transformer-based language models.
  arXiv  Detail & Related papers  (2023-03-15T00:23:49Z)
• Measure More, Question More: Experimental Studies on Transformer-based Language Models and Complement Coercion [2.1320960069210484]
  Transformer-based language models have shown strong performance on an array of natural language understanding tasks. We investigate how these models react to implicit meaning using the complement coercion phenomenon.
  arXiv  Detail & Related papers  (2022-12-20T18:46:20Z)
• Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables [17.57873577962635]
  We develop a topic-informed discrete latent variable model for semantic textual similarity. Our model learns a shared latent space for sentence-pair representation via vector quantization. We show that our model is able to surpass several strong neural baselines in semantic textual similarity tasks.
  arXiv  Detail & Related papers  (2022-11-07T15:09:58Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes.<n>We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• WMDecompose: A Framework for Leveraging the Interpretable Properties of Word Mover's Distance in Sociocultural Analysis [0.0]
  One popular model that balances legibility and interpretability is Word Mover's Distance (WMD) We introduce WMDecompose: a model and Python library that decomposes document-level distances into their constituent word-level distances, and subsequently clusters words to induce thematic elements.
  arXiv  Detail & Related papers  (2021-10-14T13:04:38Z)
• Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations [2.041108289731398]
  Recent research has adopted a new experimental field centered around the concept of text perturbations. Recent research has revealed that shuffled word order has little to no impact on the downstream performance of Transformer-based language models.
  arXiv  Detail & Related papers  (2021-09-28T20:15:29Z)
• Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge [59.22170796793179]
  Transformers Language Models (TLMs) were tested on a benchmark for the textitdynamic estimation of thematic fit Our results show that TLMs can reach performances that are comparable to those achieved by SDM. However, additional analysis consistently suggests that TLMs do not capture important aspects of event knowledge.
  arXiv  Detail & Related papers  (2021-07-22T20:52:26Z)
• Understanding Neural Abstractive Summarization Models via Uncertainty [54.37665950633147]
  seq2seq abstractive summarization models generate text in a free-form manner. We study the entropy, or uncertainty, of the model's token-level predictions. We show that uncertainty is a useful perspective for analyzing summarization and text generation models more broadly.
  arXiv  Detail & Related papers  (2020-10-15T16:57:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.