DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuation

• URL: http://arxiv.org/abs/2512.19012v2
• Date: Tue, 23 Dec 2025 06:23:22 GMT
• Title: DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuation
• Abstract summary: Drama script continuation requires models to maintain character consistency, advance plot coherently, and preserve dramatic structurecapabilities.<n>We present DramaBench, the first large-scale benchmark for evaluating drama script continuation across six independent dimensions.
• Score: 3.496854427764583
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Drama script continuation requires models to maintain character consistency, advance plot coherently, and preserve dramatic structurecapabilities that existing benchmarks fail to evaluate comprehensively. We present DramaBench, the first large-scale benchmark for evaluating drama script continuation across six independent dimensions: Format Standards, Narrative Efficiency, Character Consistency, Emotional Depth, Logic Consistency, and Conflict Handling. Our framework combines rulebased analysis with LLM-based labeling and statistical metrics, ensuring objective and reproducible evaluation. We conduct comprehensive evaluation of 8 state-of-the-art language models on 1,103 scripts (8,824 evaluations total), with rigorous statistical significance testing (252 pairwise comparisons, 65.9% significant) and human validation (188 scripts, substantial agreement on 3/5 dimensions). Our ablation studies confirm all six dimensions capture independent quality aspects (mean | r | = 0.020). DramaBench provides actionable, dimensionspecific feedback for model improvement and establishes a rigorous standard for creative writing evaluation.

Related papers

• SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions [6.2431177303574295]
  evaluating literary quality requires assessing interpretive dimensions such as cultural representation, emotional depth, and philosophical sophistication.<n>We introduce SAGE, a hierarchical evaluation framework that decomposes literary quality into straightforward interpretive dimensions.
  arXiv  Detail & Related papers  (2026-05-08T01:30:02Z)
• Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework [51.56484100374058]
  This work proposes a quantitative approach to narrative assessment by focusing on the linguistic dimension as a primary indicator of quality.<n>The paper presents a methodology for the automatic evaluation of narrative based on the extraction of a comprehensive set of 33 quantitative linguistic features.
  arXiv  Detail & Related papers  (2026-04-21T09:21:40Z)
• Revisiting Northrop Frye's Four Myths Theory with Large Language Models [0.0]
  Northrop Frye's theory of four fundamental narrative genres has profoundly influenced literary criticism.<n>We present a new character function framework that complements pattern-based analysis.<n>We derive four universal character functions (protagonist, mentor, antagonist, companion) by mapping them to Jung's psychic structure components.
  arXiv  Detail & Related papers  (2026-02-17T16:02:52Z)
• Comparative Study of Large Language Models on Chinese Film Script Continuation: An Empirical Analysis Based on GPT-5.2 and Qwen-Max [6.2773052078641385]
  This study constructs the first Chinese film script continuation benchmark comprising 53 classic films.<n> Evaluation integrates ROUGE-L, Structural Similarity, and LLM-as-Judge scoring.<n>GPT-5.2 excels in character consistency, tone-style matching, and format preservation, while Qwen-Max shows deficiencies in generation stability.
  arXiv  Detail & Related papers  (2026-01-21T09:55:26Z)
• What Matters in Evaluating Book-Length Stories? A Systematic Study of Long Story Evaluation [59.626962970198434]
  We introduce the first large-scale benchmark, LongStoryEval, comprising 600 newly published books with an average length of 121K tokens (maximum 397K)<n>By analyzing all user-mentioned aspects, we propose an evaluation criteria structure and conduct experiments to identify the most significant aspects.<n>For evaluation methods, we compare the effectiveness of three types: aggregation-based, incremental-updated, and summary-based evaluations.
  arXiv  Detail & Related papers  (2025-12-14T20:53:29Z)
• Context-Aware Pseudo-Label Scoring for Zero-Shot Video Summarization [6.057968525653529]
  We propose a rubric-guided, pseudo-labeled, and prompt-driven zero-shot video summarization framework.<n>A small subset of human annotations is converted into high-confidence pseudo labels.<n>During inference, boundary scenes are scored independently based on their own descriptions.
  arXiv  Detail & Related papers  (2025-10-20T12:54:32Z)
• ViStoryBench: Comprehensive Benchmark Suite for Story Visualization [23.274981415638837]
  ViStoryBench is a comprehensive benchmark designed to evaluate story visualization models across diverse narrative structures, visual styles, and character settings.<n>The benchmark features richly annotated multi-shot scripts derived from curated stories spanning literature, film, and folklore.<n>To enable thorough evaluation, ViStoryBench introduces a set of automated metrics that assess character consistency, style similarity, prompt adherence, aesthetic quality, and generation artifacts.
  arXiv  Detail & Related papers  (2025-05-30T17:58:21Z)
• Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings [36.449658676568234]
  Large language model (LLM)-as-judge paradigm has been used to meet the demand for a cheap, reliable, and fast evaluation of model outputs.<n>We propose ContextualJudgeBench, a judge benchmark with 2,000 challenging response pairs across eight splits inspired by real-world contextual evaluation scenarios.<n>Our comprehensive study reveals that the contextual information and its assessment criteria present a significant challenge to even state-of-the-art models.
  arXiv  Detail & Related papers  (2025-03-19T18:09:19Z)
• CharacterBench: Benchmarking Character Customization of Large Language Models [80.29164862682063]
  We propose CharacterBench, the largest bilingual generative benchmark, with 22,859 human-annotated samples covering 3,956 characters.<n>We define 11 dimensions of 6 aspects, classified as sparse and dense dimensions based on whether character features evaluated by specific dimensions manifest in each response.<n>We also develop CharacterJudge model for cost-effective and stable evaluations.
  arXiv  Detail & Related papers  (2024-12-16T15:55:34Z)
• Large Language Models are Diverse Role-Players for Summarization Evaluation [82.31575622685902]
  A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal. Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions. We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
  arXiv  Detail & Related papers  (2023-03-27T10:40:59Z)
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning [63.77667876176978]
  Large language models show improved downstream task interpretability when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness is difficult. We present ROS, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics.
  arXiv  Detail & Related papers  (2022-12-15T15:52:39Z)
• Analyzing and Evaluating Faithfulness in Dialogue Summarization [67.07947198421421]
  We first perform the fine-grained human analysis on the faithfulness of dialogue summaries and observe that over 35% of generated summaries are faithfully inconsistent respective the source dialogues. We present a new model-level faithfulness evaluation method. It examines generation models with multi-choice questions created by rule-based transformations.
  arXiv  Detail & Related papers  (2022-10-21T07:22:43Z)
• Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries [59.27273928454995]
  Current pre-trained models applied to summarization are prone to factual inconsistencies which misrepresent the source text or introduce extraneous information. We create a crowdsourcing evaluation framework for factual consistency using the rating-based Likert scale and ranking-based Best-Worst Scaling protocols. We find that ranking-based protocols offer a more reliable measure of summary quality across datasets, while the reliability of Likert ratings depends on the target dataset and the evaluation design.
  arXiv  Detail & Related papers  (2021-09-19T19:05:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.