Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework

• URL: http://arxiv.org/abs/2604.19261v1
• Date: Tue, 21 Apr 2026 09:21:40 GMT
• Title: Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework
• Abstract summary: This work proposes a quantitative approach to narrative assessment by focusing on the linguistic dimension as a primary indicator of quality.<n>The paper presents a methodology for the automatic evaluation of narrative based on the extraction of a comprehensive set of 33 quantitative linguistic features.
• Score: 51.56484100374058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The evaluation of narrative quality remains a complex challenge, as it involves subjective factors such as plot, character development, and emotional impact. This work proposes a quantitative approach to narrative assessment by focusing on the linguistic dimension as a primary indicator of quality. The paper presents a methodology for the automatic evaluation of narrative based on the extraction of a comprehensive set of 33 quantitative linguistic features categorized into lexical, syntactic, and semantic groups. To test the model, an experiment was conducted on a specialized corpus of 23 books, including canonical masterpieces and self-published works. Through a similarity matrix, the system successfully clustered the narratives, distinguishing almost perfectly between professionally edited and self-published texts. Furthermore, the methodology was validated against a human-annotated dataset; it significantly outperforms traditional story-level evaluation metrics, demonstrating the effectiveness of quantitative linguistic features in assessing narrative quality.

Related papers

• Enhancing Essay Cohesion Assessment: A Novel Item Response Theory Approach [0.7845950813414773]
  This work proposes and analyses the performance of a cohesion score prediction approach based on item response theory.<n>The proposed approach outperforms conventional machine learning models and ensemble methods in several evaluation metrics.
  arXiv  Detail & Related papers  (2025-07-11T11:05:27Z)
• What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation [57.550045763103334]
  evaluating a story can be more challenging than other generation evaluation tasks. We first summarize existing storytelling tasks, including text-to-text, visual-to-text, and text-to-visual. We propose a taxonomy to organize evaluation metrics that have been developed or can be adopted for story evaluation.
  arXiv  Detail & Related papers  (2024-08-26T20:35:42Z)
• BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence [18.77248934443666]
  Coherent texts inherently manifest a sequential and cohesive interplay among sentences.<n>BBScore is a reference-free metric grounded in Brownian bridge theory for assessing text coherence.
  arXiv  Detail & Related papers  (2023-12-28T08:34:17Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• Multi-Dimensional Evaluation of Text Summarization with In-Context Learning [79.02280189976562]
  In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning. Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization. We then analyze the effects of factors such as the selection and number of in-context examples on performance.
  arXiv  Detail & Related papers  (2023-06-01T23:27:49Z)
• Large Language Models are Diverse Role-Players for Summarization Evaluation [82.31575622685902]
  A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal. Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions. We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
  arXiv  Detail & Related papers  (2023-03-27T10:40:59Z)
• DeltaScore: Fine-Grained Story Evaluation with Perturbations [69.33536214124878]
  We introduce DELTASCORE, a novel methodology that employs perturbation techniques for the evaluation of nuanced story aspects. Our central proposition posits that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations. We measure the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models.
  arXiv  Detail & Related papers  (2023-03-15T23:45:54Z)
• M-SENSE: Modeling Narrative Structure in Short Personal Narratives Using Protagonist's Mental Representations [14.64546899992196]
  We propose the task of automatically detecting prominent elements of the narrative structure by analyzing the role of characters' inferred mental state. We introduce a STORIES dataset of short personal narratives containing manual annotations of key elements of narrative structure, specifically climax and resolution. Our model is able to achieve significant improvements in the task of identifying climax and resolution.
  arXiv  Detail & Related papers  (2023-02-18T20:48:02Z)
• Optical character recognition quality affects perceived usefulness of historical newspaper clippings [0.6299766708197884]
  Thirty-two users made searches to an article collection of Finnish newspaper Uusi Suometar 1869-1918. Our article search database had two versions of each article with different quality optical character recognition.
  arXiv  Detail & Related papers  (2022-06-01T10:07:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.