NarraBench: A Comprehensive Framework for Narrative Benchmarking

• URL: http://arxiv.org/abs/2510.09869v2
• Date: Fri, 17 Oct 2025 15:12:57 GMT
• Title: NarraBench: A Comprehensive Framework for Narrative Benchmarking
• Abstract summary: We present NarraBench, a theory-informed taxonomy of narrative-understanding tasks.<n>We estimate that only 27% of narrative tasks are well captured by existing benchmarks.
• Score: 2.2678171099418525
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present NarraBench, a theory-informed taxonomy of narrative-understanding tasks, as well as an associated survey of 78 existing benchmarks in the area. We find significant need for new evaluations covering aspects of narrative understanding that are either overlooked in current work or are poorly aligned with existing metrics. Specifically, we estimate that only 27% of narrative tasks are well captured by existing benchmarks, and we note that some areas -- including narrative events, style, perspective, and revelation -- are nearly absent from current evaluations. We also note the need for increased development of benchmarks capable of assessing constitutively subjective and perspectival aspects of narrative, that is, aspects for which there is generally no single correct answer. Our taxonomy, survey, and methodology are of value to NLP researchers seeking to test LLM narrative understanding.

Related papers

• Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey [5.628092290410201]
  This survey examines how natural language processing (NLP) research engages with fields of narrative studies.<n>We propose a taxonomy for ongoing efforts that reflect established distinctions in narratology.
  arXiv  Detail & Related papers  (2026-01-23T23:30:42Z)
• OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment [63.662126457336534]
  OpenNovelty is an agentic system for transparent, evidence-based novelty analysis.<n>It grounds all assessments in retrieved real papers, ensuring verifiable judgments.<n>OpenNovelty aims to empower the research community with a scalable tool that promotes fair, consistent, and evidence-backed peer review.
  arXiv  Detail & Related papers  (2026-01-04T15:48:51Z)
• What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation [59.626962970198434]
  We introduce the first large-scale benchmark, LongStoryEval, comprising 600 newly published books with an average length of 121K tokens (maximum 397K)<n>By analyzing all user-mentioned aspects, we propose an evaluation criteria structure and conduct experiments to identify the most significant aspects.<n>For evaluation methods, we compare the effectiveness of three types: aggregation-based, incremental-updated, and summary-based evaluations.
  arXiv  Detail & Related papers  (2025-12-14T20:53:29Z)
• Improving Narrative Classification and Explanation via Fine Tuned Language Models [6.310433217813068]
  This study tackles two key challenges: (1) multi-label classification of narratives and sub-narratives in news articles, and (2) generating concise, evidence-based explanations for dominant narratives.<n>We fine-tune a BERT model with a recall-oriented approach for comprehensive narrative detection, refining predictions using a GPT-4o pipeline for consistency.<n>Our results show that integrating auxiliary knowledge in prompts improves classification accuracy and justification reliability, with applications in media analysis, education, and intelligence gathering.
  arXiv  Detail & Related papers  (2025-09-04T10:12:31Z)
• Agent-as-Judge for Factual Summarization of Long Narratives [37.46772108810212]
  Large Language Models (LLMs) have demonstrated near-human performance in summarization tasks based on traditional metrics such as ROUGE and BERTScore.<n>Recent advances, such as LLM-as-a-Judge, address the limitations of metrics based on lexical similarity but still exhibit factual inconsistencies.<n>In this work, we introduce NarrativeFactScore, a novel "Agent-as-a-Judge" framework for evaluating and refining summaries.
  arXiv  Detail & Related papers  (2025-01-17T07:23:06Z)
• BookWorm: A Dataset for Character Description and Analysis [59.186325346763184]
  We define two tasks: character description, which generates a brief factual profile, and character analysis, which offers an in-depth interpretation. We introduce the BookWorm dataset, pairing books from the Gutenberg Project with human-written descriptions and analyses. Our findings show that retrieval-based approaches outperform hierarchical ones in both tasks.
  arXiv  Detail & Related papers  (2024-10-14T10:55:58Z)
• What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation [57.550045763103334]
  evaluating a story can be more challenging than other generation evaluation tasks. We first summarize existing storytelling tasks, including text-to-text, visual-to-text, and text-to-visual. We propose a taxonomy to organize evaluation metrics that have been developed or can be adopted for story evaluation.
  arXiv  Detail & Related papers  (2024-08-26T20:35:42Z)
• Branching Narratives: Character Decision Points Detection [13.615681132633561]
  We propose a novel dataset based on CYOA-like games graphs to be used as a benchmark for such a task. We show how such a model can be applied to the existing text to produce linear segments divided by potential branching points.
  arXiv  Detail & Related papers  (2024-05-12T13:36:07Z)
• Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions [48.18584733906447]
  This work introduces an original and practical paradigm for narrative comprehension, stemming from the characteristics that individual passages within narratives tend to be more cohesively related than isolated. We propose a fine-grained modeling of narrative context, by formulating a graph dubbed NarCo, which explicitly depicts task-agnostic coherence dependencies.
  arXiv  Detail & Related papers  (2024-02-21T06:14:04Z)
• DeltaScore: Fine-Grained Story Evaluation with Perturbations [69.33536214124878]
  We introduce DELTASCORE, a novel methodology that employs perturbation techniques for the evaluation of nuanced story aspects. Our central proposition posits that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations. We measure the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models.
  arXiv  Detail & Related papers  (2023-03-15T23:45:54Z)
• Screenplay Summarization Using Latent Narrative Structure [78.45316339164133]
  We propose to explicitly incorporate the underlying structure of narratives into general unsupervised and supervised extractive summarization models. We formalize narrative structure in terms of key narrative events (turning points) and treat it as latent in order to summarize screenplays. Experimental results on the CSI corpus of TV screenplays, which we augment with scene-level summarization labels, show that latent turning points correlate with important aspects of a CSI episode.
  arXiv  Detail & Related papers  (2020-04-27T11:54:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.