Surprisal Takes It All: Eye Tracking Based Cognitive Evaluation of Text Readability Measures

• URL: http://arxiv.org/abs/2502.11150v1
• Date: Sun, 16 Feb 2025 14:51:44 GMT
• Title: Surprisal Takes It All: Eye Tracking Based Cognitive Evaluation of Text Readability Measures
• Authors: Keren Gruteke Klein, Shachar Frenkel, Omer Shubi, Yevgeni Berzak,
• Abstract summary: We propose a new eye tracking based methodology for evaluating readability measures. We find that existing readability formulas are moderate to poor predictors of reading ease. Average per-word length, frequency, and especially surprisal tend to outperform existing readability formulas as measures of reading ease.
• Score: 1.2062053320259833
• License:
• Abstract: Text readability measures are widely used in many real-world scenarios and in NLP. These measures have primarily been developed by predicting reading comprehension outcomes, while largely neglecting what is perhaps the core aspect of a readable text: reading ease. In this work, we propose a new eye tracking based methodology for evaluating readability measures, which focuses on their ability to account for reading facilitation effects in text simplification, as well as for text reading ease more broadly. Using this approach, we find that existing readability formulas are moderate to poor predictors of reading ease. We further find that average per-word length, frequency, and especially surprisal tend to outperform existing readability formulas as measures of reading ease. We thus propose surprisal as a simple unsupervised alternative to existing measures.

Related papers

• A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attribution [57.309390098903]
  Authorship attribution aims to identify the origin or author of a document. Large Language Models (LLMs) with their deep reasoning capabilities and ability to maintain long-range textual associations offer a promising alternative. Our results on the IMDb and blog datasets show an impressive 85% accuracy in one-shot authorship classification across ten authors.
  arXiv  Detail & Related papers  (2024-10-29T04:14:23Z)
• Free-text Rationale Generation under Readability Level Control [6.338124510580766]
  We investigate how large language models (LLMs) perform rationale generation under the effects of readability level control. We find that explanations are adaptable to such instruction, though the requested readability is often misaligned with the measured text complexity. Our human annotators confirm a generally satisfactory impression on rationales at all readability levels.
  arXiv  Detail & Related papers  (2024-07-01T15:34:17Z)
• Previously on the Stories: Recap Snippet Identification for Story Reading [51.641565531840186]
  We propose the first benchmark on this useful task called Recap Snippet Identification with a hand-crafted evaluation dataset. Our experiments show that the proposed task is challenging to PLMs, LLMs, and proposed methods as the task requires a deep understanding of the plot correlation between snippets.
  arXiv  Detail & Related papers  (2024-02-11T18:27:14Z)
• Generating Summaries with Controllable Readability Levels [67.34087272813821]
  Several factors affect the readability level, such as the complexity of the text, its subject matter, and the reader's background knowledge. Current text generation approaches lack refined control, resulting in texts that are not customized to readers' proficiency levels. We develop three text generation techniques for controlling readability: instruction-based readability control, reinforcement learning to minimize the gap between requested and observed readability, and a decoding approach that uses look-ahead to estimate the readability of upcoming decoding steps.
  arXiv  Detail & Related papers  (2023-10-16T17:46:26Z)
• LC-Score: Reference-less estimation of Text Comprehension Difficulty [0.0]
  We present textscLC-Score, a simple approach for training text comprehension metric for any French text without reference. Our objective is to quantitatively capture the extend to which a text suits to the textitLangage Clair (LC, textitClear Language) guidelines. We explore two approaches: (i) using linguistically motivated indicators used to train statistical models, and (ii) neural learning directly from text leveraging pre-trained language models.
  arXiv  Detail & Related papers  (2023-10-04T11:49:37Z)
• Self-Verification Improves Few-Shot Clinical Information Extraction [73.6905567014859]
  Large language models (LLMs) have shown the potential to accelerate clinical curation via few-shot in-context learning. They still struggle with issues regarding accuracy and interpretability, especially in mission-critical domains such as health. Here, we explore a general mitigation framework using self-verification, which leverages the LLM to provide provenance for its own extraction and check its own outputs.
  arXiv  Detail & Related papers  (2023-05-30T22:05:11Z)
• Readability Research: An Interdisciplinary Approach [62.03595526230364]
  We aim to provide a firm foundation for readability research, a comprehensive framework for readability research. Readability refers to aspects of visual information design which impact information flow from the page to the reader. These aspects can be modified on-demand, instantly improving the ease with which a reader can process and derive meaning from text.
  arXiv  Detail & Related papers  (2021-07-20T16:52:17Z)
• On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach [3.2326259807823026]
  We analyze an alternative PMI-based metric to quantify biases in texts. It can be expressed as a function of conditional probabilities, which provides a simple interpretation in terms of word co-occurrences.
  arXiv  Detail & Related papers  (2021-04-13T19:34:17Z)
• Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning [94.50608198582636]
  Interactive Fiction (IF) games with real human-written natural language texts provide a new natural evaluation for language understanding techniques. We take a novel perspective of IF game solving and re-formulate it as Multi-Passage Reading (MPRC) tasks.
  arXiv  Detail & Related papers  (2020-10-05T23:09:20Z)
• Text as Environment: A Deep Reinforcement Learning Text Readability Assessment Model [2.826553192869411]
  The efficiency of state-of-the-art text readability assessment models can be further improved using deep reinforcement learning models. A comparison of the model on Weebit and Cambridge Exams with state-of-the-art models, such as the BERT text readability model, shows that it is capable of achieving state-of-the-art accuracy with a significantly smaller amount of input text than other models.
  arXiv  Detail & Related papers  (2019-12-12T13:54:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.