On the Proper Treatment of Units in Surprisal Theory

• URL: http://arxiv.org/abs/2604.28147v1
• Date: Thu, 30 Apr 2026 17:33:58 GMT
• Title: On the Proper Treatment of Units in Surprisal Theory
• Abstract summary: We show that surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices.<n>We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail.
• Score: 50.52310749207036
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive.

Related papers

• Formal Abductive Latent Explanations for Prototype-Based Networks [7.001970497421476]
  Case-based reasoning networks make predictions based on similarity between the input and prototypical parts of training samples, called prototypes.<n>We show that such explanations are sometimes misleading, which hampers their usefulness in safety-critical contexts.<n>We propose Abductive Latent Explanations (ALEs), a formalism to express sufficient conditions on the intermediate representation of the instance that imply the prediction.
  arXiv  Detail & Related papers  (2025-11-20T17:42:41Z)
• Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors [61.92704516732144]
  We show that the most robust features for correctness prediction are those that play a distinctive causal role in the model's behavior.<n>We propose two methods that leverage causal mechanisms to predict the correctness of model outputs.
  arXiv  Detail & Related papers  (2025-05-17T00:31:39Z)
• Invariant Causal Set Covering Machines [48.169632766444906]
  Rule-based models, such as decision trees, appeal to practitioners due to their interpretable nature.<n>However, the learning algorithms that produce such models are often vulnerable to spurious associations and thus, they are not guaranteed to extract causally-relevant insights.<n>We propose Invariant Causal Set Covering Machines, an extension of the classical Set Covering Machine algorithm for conjunctions/disjunctions of binary-valued rules that provably avoids spurious associations.
  arXiv  Detail & Related papers  (2023-06-07T20:52:01Z)
• Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions [9.909170013118775]
  This work presents a linear decomposition of final hidden states from autoregressive language models based on each initial input token. Using the change in next-word probability as a measure of importance, this work first examines which context words make the biggest contribution to language model predictions.
  arXiv  Detail & Related papers  (2023-05-17T23:55:32Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes.<n>We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Instance-Based Neural Dependency Parsing [56.63500180843504]
  We develop neural models that possess an interpretable inference process for dependency parsing. Our models adopt instance-based inference, where dependency edges are extracted and labeled by comparing them to edges in a training set.
  arXiv  Detail & Related papers  (2021-09-28T05:30:52Z)
• Discrete representations in neural models of spoken language [56.29049879393466]
  We compare the merits of four commonly used metrics in the context of weakly supervised models of spoken language. We find that the different evaluation metrics can give inconsistent results.
  arXiv  Detail & Related papers  (2021-05-12T11:02:02Z)
• On the Branching Bias of Syntax Extracted from Pre-trained Language Models [47.82102426290707]
  We propose quantitatively measuring the branching bias by comparing the performance gap on a language and its reversed language. We analyze the impacts of three factors on the branching bias, namely parsing algorithms, feature definitions, and language models.
  arXiv  Detail & Related papers  (2020-10-06T03:09:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.