Lost in Stories: Consistency Bugs in Long Story Generation by LLMs

• URL: http://arxiv.org/abs/2603.05890v1
• Date: Fri, 06 Mar 2026 04:16:46 GMT
• Title: Lost in Stories: Consistency Bugs in Long Story Generation by LLMs
• Abstract summary: ConStory-Bench is a benchmark designed to evaluate narrative consistency in long-form story generation.<n>It contains 2,000 prompts across four task scenarios and defines a taxonomy of five error categories with 19 fine-grained subtypes.<n>We also develop ConStory-Checker, an automated pipeline that detects contradictions and grounds each judgment in explicit textual evidence.
• Score: 21.241285394393632
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: What happens when a storyteller forgets its own story? Large Language Models (LLMs) can now generate narratives spanning tens of thousands of words, but they often fail to maintain consistency throughout. When generating long-form narratives, these models can contradict their own established facts, character traits, and world rules. Existing story generation benchmarks focus mainly on plot quality and fluency, leaving consistency errors largely unexplored. To address this gap, we present ConStory-Bench, a benchmark designed to evaluate narrative consistency in long-form story generation. It contains 2,000 prompts across four task scenarios and defines a taxonomy of five error categories with 19 fine-grained subtypes. We also develop ConStory-Checker, an automated pipeline that detects contradictions and grounds each judgment in explicit textual evidence. Evaluating a range of LLMs through five research questions, we find that consistency errors show clear tendencies: they are most common in factual and temporal dimensions, tend to appear around the middle of narratives, occur in text segments with higher token-level entropy, and certain error types tend to co-occur. These findings can inform future efforts to improve consistency in long-form narrative generation. Our project page is available at https://picrew.github.io/constory-bench.github.io/.

Related papers

• From Personas to Plot: Character-Grounded Multi-Agent Story Generation for Long-Form Narratives [3.0130431031064613]
  We introduce a unified framework for long-form narrative generation and verification.<n>We show that our framework produces coherent narratives compared to single-model prompting and IBSEN.<n>These results suggest that long-form narratives can emerge from explicit world-state tracking and goal-driven multi-agent generation.
  arXiv  Detail & Related papers  (2026-07-01T13:23:07Z)
• Do Large Language Models Always Tell The Same Stories? [2.4779770815686444]
  We investigate the diversity of large language models-generated stories through the framework of narrative similarity.<n>Our findings reveal a consistent trend: LLM-generated narratives are consistently more similar to each other than human-written stories are.<n>We show that common mitigation strategies, including negative prompting and temperature scaling, fail to meaningfully address this homogeneity.
  arXiv  Detail & Related papers  (2026-06-15T22:52:02Z)
• Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling [15.25806708314033]
  We argue that existing rubrics overlook a key dimension of compelling human stories: narrative tension.<n>We introduce the 100-Endings metric, which walks through a story sentence by sentence.<n>Unlike rubric-based judges, 100-Endings correctly ranks New Yorker stories far above LLM outputs.
  arXiv  Detail & Related papers  (2026-04-10T19:33:43Z)
• Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novels [50.43968216132018]
  We present an end-to-end system that transforms any literary work into an immersive, multi-character conversational experience.<n>This system is designed to solve two fundamental challenges for LLM-driven characters.
  arXiv  Detail & Related papers  (2025-12-08T11:57:46Z)
• NOAH: Benchmarking Narrative Prior driven Hallucination and Omission in Video Large Language Models [8.6767620170781]
  Video large language models (Video LLMs) have recently achieved strong performance on tasks such as captioning, summarization, and question answering.<n>Many models and training methods explicitly encourage continuity across events to enhance narrative coherence.<n>We identify this bias, which we call narrative prior, as a key driver of two errors: hallucinations, where non-existent events are introduced or existing ones are misinterpreted, and omissions, where factual events are suppressed because they are misaligned with surrounding context.
  arXiv  Detail & Related papers  (2025-11-09T17:41:11Z)
• STORYANCHORS: Generating Consistent Multi-Scene Story Frames for Long-Form Narratives [82.19488717416351]
  This paper introduces StoryAnchors, a unified framework for generating high-quality, multi-scene story frames.<n>StoryAnchors employs a bidirectional story generator that integrates both past and future contexts to ensure temporal consistency.<n>It also integrates Multi-Event Story Frame Labeling and Progressive Story Frame Training, enabling the model to capture both overarching narrative flow and event-level dynamics.
  arXiv  Detail & Related papers  (2025-05-13T08:48:10Z)
• Generating Visual Stories with Grounded and Coreferent Characters [63.07511918366848]
  We present the first model capable of predicting visual stories with consistently grounded and coreferent character mentions.<n>Our model is finetuned on a new dataset which we build on top of the widely used VIST benchmark.<n>We also propose new evaluation metrics to measure the richness of characters and coreference in stories.
  arXiv  Detail & Related papers  (2024-09-20T14:56:33Z)
• Re3: Generating Longer Stories With Recursive Reprompting and Revision [83.99558005056817]
  We consider the problem of automatically generating longer stories of over two thousand words. Compared to prior work on shorter stories, long-range plot coherence and relevance are more central challenges here. We propose the Recursive Reprompting and Revision framework (Re3) to address these challenges.
  arXiv  Detail & Related papers  (2022-10-13T06:29:57Z)
• Computational Lens on Cognition: Study Of Autobiographical Versus Imagined Stories With Large-Scale Language Models [95.88620740809004]
  We study differences in the narrative flow of events in autobiographical versus imagined stories using GPT-3. We found that imagined stories have higher sequentiality than autobiographical stories. In comparison to imagined stories, autobiographical stories contain more concrete words and words related to the first person.
  arXiv  Detail & Related papers  (2022-01-07T20:10:47Z)
• Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning [12.264880519328353]
  We introduce Commonsense-inference Augmented neural StoryTelling (CAST), a framework for introducing commonsense reasoning into the generation process. We find that our CAST method produces significantly more coherent, on-topic, enjoyable and fluent stories than existing models in both the single-character and two-character settings.
  arXiv  Detail & Related papers  (2021-05-04T06:40:33Z)
• Consistency and Coherency Enhanced Story Generation [35.08911595854691]
  We propose a two-stage generation framework to enhance consistency and coherency of generated stories. The first stage is to organize the story outline which depicts the story plots and events, and the second stage is to expand the outline into a complete story. In addition, coreference supervision signals are incorporated to reduce coreference errors and improve the coreference consistency.
  arXiv  Detail & Related papers  (2020-10-17T16:40:37Z)
• UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation [92.42032403795879]
  UNION is a learnable unreferenced metric for evaluating open-ended story generation. It is trained to distinguish human-written stories from negative samples and recover the perturbation in negative stories. Experiments on two story datasets demonstrate that UNION is a reliable measure for evaluating the quality of generated stories.
  arXiv  Detail & Related papers  (2020-09-16T11:01:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.