Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives

• URL: http://arxiv.org/abs/2310.13676v1
• Date: Fri, 20 Oct 2023 17:25:36 GMT
• Title: Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives
• Authors: Mario Giulianelli, Sarenne Wallbridge, Raquel Fern\'andez
• Abstract summary: We present information value, a measure which quantifies the predictability of an utterance relative to a set of plausible alternatives. We exploit their psychometric predictive power to investigate the dimensions of predictability that drive human comprehension behaviour.
• Score: 4.446323294830542
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present information value, a measure which quantifies the predictability of an utterance relative to a set of plausible alternatives. We introduce a method to obtain interpretable estimates of information value using neural text generators, and exploit their psychometric predictive power to investigate the dimensions of predictability that drive human comprehension behaviour. Information value is a stronger predictor of utterance acceptability in written and spoken dialogue than aggregates of token-level surprisal and it is complementary to surprisal for predicting eye-tracked reading times.

Related papers

• Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Measures [1.2062053320259833]
  We propose an eye tracking-based cognitive framework which taps into a key aspect of readability: reading ease.<n>We use this framework for evaluating a broad range of prominent readability measures, including two systems widely used in education.<n>Our analyses suggest that existing readability measures are poor predictors of reading facilitation and reading ease, outperformed by word properties commonly used in psycholinguistics.
  arXiv  Detail & Related papers  (2025-02-16T14:51:44Z)
• XForecast: Evaluating Natural Language Explanations for Time Series Forecasting [72.57427992446698]
  Time series forecasting aids decision-making, especially for stakeholders who rely on accurate predictions. Traditional explainable AI (XAI) methods, which underline feature or temporal importance, often require expert knowledge. evaluating forecast NLEs is difficult due to the complex causal relationships in time series data.
  arXiv  Detail & Related papers  (2024-10-18T05:16:39Z)
• Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences [1.942809872918085]
  We revisit the predictive power of surprisal and entropy measures estimated from a range of language models (LMs) on data of human reading times. We investigate if modulating surprisal and entropy relative to cognitive scores increases prediction accuracy of reading times. Our study finds that in most cases, incorporating cognitive capacities increases predictive power of surprisal and entropy on reading times.
  arXiv  Detail & Related papers  (2024-06-07T14:54:56Z)
• Quantifying the Plausibility of Context Reliance in Neural Machine Translation [25.29330352252055]
  We introduce Plausibility Evaluation of Context Reliance (PECoRe) PECoRe is an end-to-end interpretability framework designed to quantify context usage in language models' generations. We use pecore to quantify the plausibility of context-aware machine translation models.
  arXiv  Detail & Related papers  (2023-10-02T13:26:43Z)
• Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
  We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
  arXiv  Detail & Related papers  (2023-08-03T12:43:21Z)
• Probabilistic Prompt Learning for Dense Prediction [45.577125507777474]
  We present a novel probabilistic prompt learning to fully exploit the vision-language knowledge in dense prediction tasks. We introduce learnable class-agnostic attribute prompts to describe universal attributes across the object class. The attributes are combined with class information and visual-context knowledge to define the class-specific textual distribution.
  arXiv  Detail & Related papers  (2023-04-03T08:01:27Z)
• Prediction-Powered Inference [68.97619568620709]
  Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system. The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients. Prediction-powered inference could enable researchers to draw valid and more data-efficient conclusions using machine learning.
  arXiv  Detail & Related papers  (2023-01-23T18:59:28Z)
• What Should I Know? Using Meta-gradient Descent for Predictive Feature Discovery in a Single Stream of Experience [63.75363908696257]
  computational reinforcement learning seeks to construct an agent's perception of the world through predictions of future sensations. An open challenge in this line of work is determining from the infinitely many predictions that the agent could possibly make which predictions might best support decision-making. We introduce a meta-gradient descent process by which an agent learns what predictions to make, 2) the estimates for its chosen predictions, and 3) how to use those estimates to generate policies that maximize future reward.
  arXiv  Detail & Related papers  (2022-06-13T21:31:06Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers [0.05857406612420462]
  Large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks. We propose evaluating systems through a novel measure of prediction coherence.
  arXiv  Detail & Related papers  (2021-09-10T15:04:23Z)
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components [96.42805872177067]
  We propose a self-supervised representation learning method for sequence data, based on the intuition that useful representations of sequence data should exhibit a simple structure in the latent space. We encourage this latent structure by maximizing an estimate of predictive information of latent feature sequences, which is the mutual information between past and future windows at each time step. We demonstrate that our method recovers the latent space of noisy dynamical systems, extracts predictive features for forecasting tasks, and improves automatic speech recognition when used to pretrain the encoder on large amounts of unlabeled data.
  arXiv  Detail & Related papers  (2020-10-07T03:34:01Z)
• Towards a Measure of Individual Fairness for Deep Learning [2.4366811507669124]
  We show how to compute prediction sensitivity using standard automatic differentiation capabilities present in modern deep learning frameworks. Preliminary empirical results suggest that prediction sensitivity may be effective for measuring bias in individual predictions.
  arXiv  Detail & Related papers  (2020-09-28T21:53:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.