Modeling Fair Play in Detective Stories with Language Models

• URL: http://arxiv.org/abs/2507.13841v1
• Date: Fri, 18 Jul 2025 11:55:18 GMT
• Title: Modeling Fair Play in Detective Stories with Language Models
• Abstract summary: We present a probabilistic framework for detective fiction that allows us to define desired qualities.<n>We validate the framework by applying it to LLM-generated detective stories.<n>Results show that while LLM-generated stories may be unpredictable, they generally fail to balance the trade-off between surprise and fair play.
• Score: 14.90042250991032
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Effective storytelling relies on a delicate balance between meeting the reader's prior expectations and introducing unexpected developments. In the domain of detective fiction, this tension is known as fair play, which includes the implicit agreement between the writer and the reader as to the range of possible resolutions the mystery story may have. In this work, we present a probabilistic framework for detective fiction that allows us to define desired qualities. Using this framework, we formally define fair play and design appropriate metrics for it. Stemming from these definitions is an inherent tension between the coherence of the story, which measures how much it ``makes sense'', and the surprise it induces. We validate the framework by applying it to LLM-generated detective stories. This domain is appealing since we have an abundance of data, we can sample from the distribution generating the story, and the story-writing capabilities of LLMs are interesting in their own right. Results show that while LLM-generated stories may be unpredictable, they generally fail to balance the trade-off between surprise and fair play, which greatly contributes to their poor quality.

Related papers

• NARRA-Gym for Evaluating Interactive Narrative Agents [69.49891044929372]
  We introduce NARRA-Gym, an evaluation environment that turns a sparse emotional seed into a complete interactive story episode.<n>We evaluate nine frontier LLMs using a controlled LLM-as-judge sweep over eight benchmark personas and a human evaluation in which participants rate customized model outputs.
  arXiv  Detail & Related papers  (2026-05-08T21:36:23Z)
• Codified Foreshadowing-Payoff Text Generation [67.01182739162142]
  Foreshadowing and payoff are ubiquitous narrative devices through which authors introduce commitments early in a story and resolve them through concrete, observable outcomes.<n>Existing evaluations largely overlook this structural failure, focusing on surface-level coherence rather than the logical fulfillment of narrative setups.<n>We introduce Codified Foreshadowing-Payoff Generation, a novel framework that reframes narrative quality through the lens of payoff realization.
  arXiv  Detail & Related papers  (2026-01-11T19:05:37Z)
• Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novels [50.43968216132018]
  We present an end-to-end system that transforms any literary work into an immersive, multi-character conversational experience.<n>This system is designed to solve two fundamental challenges for LLM-driven characters.
  arXiv  Detail & Related papers  (2025-12-08T11:57:46Z)
• Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection [35.550137361809405]
  Plot hole detection in stories is a proxy to evaluate language understanding and reasoning in Large Language Models.<n>We introduce FlawedFictionsMaker, a novel algorithm to controllably and carefully synthesize plot holes in human-written stories.<n>We find that state-of-the-art LLMs struggle in accurately solving FlawedFictions regardless of the reasoning effort allowed.
  arXiv  Detail & Related papers  (2025-04-16T09:25:54Z)
• Learning to Reason for Long-Form Story Generation [98.273323001781]
  We propose a general story-generation task (Next-Chapter Prediction) and a reward formulation (Verified Rewards via Completion Likelihood Improvement)<n>We learn to reason over a story's condensed information and generate a detailed plan for the next chapter.<n>Our reasoning is evaluated via the chapters it helps a story-generator create, and compared against non-trained and supervised finetuning (SFT) baselines.
  arXiv  Detail & Related papers  (2025-03-28T18:48:26Z)
• Show, Don't Tell: Uncovering Implicit Character Portrayal using LLMs [19.829683714192615]
  We introduce LIIPA, a framework for prompting large language models to uncover implicit character portrayals.<n>We find that LIIPA outperforms existing approaches, and is more robust to increasing character counts.<n>Our work demonstrates the potential benefits of using LLMs to analyze complex characters.
  arXiv  Detail & Related papers  (2024-12-05T19:46:53Z)
• Agents' Room: Narrative Generation through Multi-step Collaboration [54.98886593802834]
  We propose a generation framework inspired by narrative theory that decomposes narrative writing into subtasks tackled by specialized agents.<n>We show that Agents' Room generates stories preferred by expert evaluators over those produced by baseline systems.
  arXiv  Detail & Related papers  (2024-10-03T15:44:42Z)
• Generating Visual Stories with Grounded and Coreferent Characters [63.07511918366848]
  We present the first model capable of predicting visual stories with consistently grounded and coreferent character mentions.<n>Our model is finetuned on a new dataset which we build on top of the widely used VIST benchmark.<n>We also propose new evaluation metrics to measure the richness of characters and coreference in stories.
  arXiv  Detail & Related papers  (2024-09-20T14:56:33Z)
• Are Large Language Models Capable of Generating Human-Level Narratives? [114.34140090869175]
  This paper investigates the capability of LLMs in storytelling, focusing on narrative development and plot progression. We introduce a novel computational framework to analyze narratives through three discourse-level aspects. We show that explicit integration of discourse features can enhance storytelling, as is demonstrated by over 40% improvement in neural storytelling.
  arXiv  Detail & Related papers  (2024-07-18T08:02:49Z)
• LFED: A Literary Fiction Evaluation Dataset for Large Language Models [58.85989777743013]
  We collect 95 literary fictions that are either originally written in Chinese or translated into Chinese, covering a wide range of topics across several centuries. We define a question taxonomy with 8 question categories to guide the creation of 1,304 questions. We conduct an in-depth analysis to ascertain how specific attributes of literary fictions (e.g., novel types, character numbers, the year of publication) impact LLM performance in evaluations.
  arXiv  Detail & Related papers  (2024-05-16T15:02:24Z)
• Let Storytelling Tell Vivid Stories: An Expressive and Fluent Multimodal Storyteller [21.953766228135827]
  We propose a new pipeline, termed LLaMS, to generate multimodal human-level stories. We first employ a sequence data auto-enhancement strategy to enhance factual content expression. Secondly, we propose SQ-Adatpter module for story illustration generation which can maintain sequence consistency.
  arXiv  Detail & Related papers  (2024-03-12T04:07:00Z)
• Paragraph-level Commonsense Transformers with Recurrent Memory [77.4133779538797]
  We train a discourse-aware model that incorporates paragraph-level information to generate coherent commonsense inferences from narratives. Our results show that PARA-COMET outperforms the sentence-level baselines, particularly in generating inferences that are both coherent and novel.
  arXiv  Detail & Related papers  (2020-10-04T05:24:12Z)
• Modelling Suspense in Short Stories as Uncertainty Reduction over Neural Representation [17.683705809784453]
  Suspense is a crucial ingredient of narrative fiction, engaging readers and making stories compelling. We compare two ways for modelling suspense: surprise, a backward-looking measure of how unexpected the current state is given the story so far; and uncertainty reduction, a forward-looking measure of how unexpected the continuation of the story is. We propose a hierarchical language model that encodes stories and computes surprise and uncertainty reduction.
  arXiv  Detail & Related papers  (2020-04-30T16:03:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.