SAGE: A Hierarchical Framework for Evaluating Interpretive Literary Quality in Narratives

• URL: http://arxiv.org/abs/2609.06611v1
• Date: Sun, 06 Sep 2026 13:57:31 GMT
• Title: SAGE: A Hierarchical Framework for Evaluating Interpretive Literary Quality in Narratives
• Abstract summary: We introduce SAGE, a six-layer evaluation framework that separates rule-based assessment of observable properties from LLM-based evaluation of interpretive qualities.<n>Our central finding is a systematic capability boundary: emotional-psychological representation approaches human levels, while cultural critique and philosophical depth exhibit approximately double the gap.<n>We interpret this as a boundary between pattern-reproducible literary capacities learnable from training corpora and stance-requiring ones demanding cultural positioning and philosophical engagement that pattern matching alone cannot provide.
• Score: 6.2431177303574295
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Assessing the literary quality of narratives requires evaluating interpretive dimensions (cultural representation, emotional depth, and philosophical engagement) that existing NLG metrics cannot measure. We introduce SAGE, a six-layer evaluation framework that separates rule-based assessment of observable textual properties from LLM-based evaluation of interpretive qualities drawn from cultural theory, affect theory, and existentialist philosophy. Each interpretive layer is assessed through multi-round iterative LLM evaluation with independent cross-validation, achieving measurement-grade reliability (98.8% convergence, >94% inter-rater agreement) stable across evaluator models. Validated on 600 evaluations across 100 short stories, our central finding is a systematic capability boundary: emotional-psychological representation approaches human levels, while cultural critique and philosophical depth exhibit approximately double the gap. LLM-generated narratives score below even commercial genre fiction on all three layers. We interpret this as a boundary between pattern-reproducible literary capacities learnable from training corpora and stance-requiring ones demanding cultural positioning and philosophical engagement that pattern matching alone cannot provide.

Related papers

• Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference [21.264274413987575]
  PRISM (Persona Reasoning with Inverse SFL-based Modeling) is a psycholinguistically grounded framework that reformulates persona fidelity evaluation as a structured inverse inference task.<n>Inspired by Systemic Functional Linguistics (SFL), PRISM decomposes persona fidelity into three functional dimensions: Task Framing, Interpersonal Stance, and Linguistic Style.<n>Experiments show that PRISM yields more accurate and stable judgements than traditional holistic judging, providing a more reliable framework for persona fidelity evaluation.
  arXiv  Detail & Related papers  (2026-08-27T06:27:51Z)
• A Survey of Large Language Models for Perception and Measurement of Human Psychology [55.608047817598965]
  A central issue is whether Large Language Models possess the capability to accurately perceive and measure complex, latent human psychological constructs.<n>This paper provides a systematic review focused on the use of LLMs as instruments for human psychological measurement.
  arXiv  Detail & Related papers  (2026-05-20T14:54:09Z)
• SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions [6.2431177303574295]
  evaluating literary quality requires assessing interpretive dimensions such as cultural representation, emotional depth, and philosophical sophistication.<n>We introduce SAGE, a hierarchical evaluation framework that decomposes literary quality into straightforward interpretive dimensions.
  arXiv  Detail & Related papers  (2026-05-08T01:30:02Z)
• Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework [51.56484100374058]
  This work proposes a quantitative approach to narrative assessment by focusing on the linguistic dimension as a primary indicator of quality.<n>The paper presents a methodology for the automatic evaluation of narrative based on the extraction of a comprehensive set of 33 quantitative linguistic features.
  arXiv  Detail & Related papers  (2026-04-21T09:21:40Z)
• Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective [4.313916831146957]
  In cognitive science and linguistic theory, dialogue is not a chain of independent utterances but a joint activity sustained by coherence, consistency, and shared understanding.<n>We re-examine a notable retrieval-augmented framework for personalized dialogue, LAPDOG, as a case study for evaluation methodology.
  arXiv  Detail & Related papers  (2026-03-15T04:36:16Z)
• Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry [4.720025219010595]
  Large Language Models (LLMs) are increasingly applied to creative domains, yet their performance in classical Chinese poetry generation and evaluation remains poorly understood.<n>We propose a three-step evaluation framework that combines computational metrics, LLM-as-a-judge assessment, and human expert validation.
  arXiv  Detail & Related papers  (2025-10-17T05:00:37Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey [64.08485471150486]
  This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings.<n>We systematically reviewed nearly 250 scholarly sources, capturing the state of the art from various venues of publication.
  arXiv  Detail & Related papers  (2025-03-28T14:08:40Z)
• Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs [7.802103248428407]
  We identify and test three assumptions behind current survey-based evaluation methods.<n>We find a high level of instability across presentation formats, incoherence between evaluated versus held-out cultural dimensions, and erratic behavior under prompt steering.
  arXiv  Detail & Related papers  (2025-03-11T17:59:53Z)
• Evaluating Large Language Models with Psychometrics [59.821829073478376]
  This paper offers a comprehensive benchmark for quantifying psychological constructs of Large Language Models (LLMs)<n>Our work identifies five key psychological constructs -- personality, values, emotional intelligence, theory of mind, and self-efficacy -- assessed through a suite of 13 datasets.<n>We uncover significant discrepancies between LLMs' self-reported traits and their response patterns in real-world scenarios, revealing complexities in their behaviors.
  arXiv  Detail & Related papers  (2024-06-25T16:09:08Z)
• Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches [69.73783026870998]
  This work proposes a novel framework, ValueLex, to reconstruct Large Language Models' unique value system from scratch. Based on Lexical Hypothesis, ValueLex introduces a generative approach to elicit diverse values from 30+ LLMs. We identify three core value dimensions, Competence, Character, and Integrity, each with specific subdimensions, revealing that LLMs possess a structured, albeit non-human, value system.
  arXiv  Detail & Related papers  (2024-04-19T09:44:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.