Characterizing Narrative Content in Web-scale LLM Pretraining Data

• URL: http://arxiv.org/abs/2606.19468v1
• Date: Wed, 17 Jun 2026 18:03:34 GMT
• Title: Characterizing Narrative Content in Web-scale LLM Pretraining Data
• Abstract summary: We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus.<n>We design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions.<n>We apply NarraBERT to 3M passages, resulting in a new dataset, NarraDolma.
• Score: 14.02632268506113
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The narrative composition of web-scale LLM pretraining corpora remains largely unexplored even though narrative is a fundamental mode of human communication. We present the first fine-grained study of narrative features in Dolma, a 3-trillion-token open pretraining corpus. Drawing on narrative theory, we design a framework spanning three core narrative elements (agency, setting, and events) operationalized as 11 interpretable dimensions. After sampling and annotating a diverse set of 400 passages, we finetune and validate NarraBERT, a RoBERTa-based model for fine-grained narrative prediction. We apply NarraBERT to 3M passages, resulting in a new dataset, NarraDolma. We find (i) narrative structure is measurable at scale across extremely heterogeneous data, (ii) we uncover a continuous, multidimensional narrative structure underlying web text, and (iii) narrative qualities are unequally distributed across pretraining sources and topics in ways that current curation practices neither measure nor account for. Our framework, dataset, and analyses provide a foundation for understanding how narrative qualities are distributed in LLM pretraining data and for studying how data composition affects narrative reasoning tasks. We publicly release NarraDolma and NarraBERT.

Related papers

• Creative Convergence or Imitation? Genre-Specific Homogeneity in LLM-Generated Chinese Literature [17.985893852336865]
  We propose a novel theoretical framework for analysis by incorporating Proppian narratology and narrative functions.<n>We extend Propp's narrative theory, defining 34 narrative functions suited to modern web narrative structures.<n> Experiments reveal that the primary reasons for the singular narrative logic and severe homogenization in generated texts are that current LLMs are unable to correctly comprehend the meanings of narrative functions.
  arXiv  Detail & Related papers  (2026-03-15T15:15:48Z)
• Classifying Unreliable Narrators with Large Language Models [23.817691955577835]
  We present TUNa, a human-annotated dataset of narratives from multiple domains.<n>We define classification tasks for intra-narrational, inter-narrational, and inter-textual unreliabilities.<n>We propose learning from literature to perform unreliable narrator classification on real-world text data.
  arXiv  Detail & Related papers  (2025-06-11T23:17:12Z)
• BookWorm: A Dataset for Character Description and Analysis [59.186325346763184]
  We define two tasks: character description, which generates a brief factual profile, and character analysis, which offers an in-depth interpretation. We introduce the BookWorm dataset, pairing books from the Gutenberg Project with human-written descriptions and analyses. Our findings show that retrieval-based approaches outperform hierarchical ones in both tasks.
  arXiv  Detail & Related papers  (2024-10-14T10:55:58Z)
• Mapping News Narratives Using LLMs and Narrative-Structured Text Embeddings [0.0]
  We introduce a numerical narrative representation grounded in structuralist linguistic theory. We extract the actants using an open-source LLM and integrate them into a Narrative-Structured Text Embedding. We demonstrate the analytical insights of the method on the example of 5000 full-text news articles from Al Jazeera and The Washington Post on the Israel-Palestine conflict.
  arXiv  Detail & Related papers  (2024-09-10T14:15:30Z)
• Are Large Language Models Capable of Generating Human-Level Narratives? [114.34140090869175]
  This paper investigates the capability of LLMs in storytelling, focusing on narrative development and plot progression. We introduce a novel computational framework to analyze narratives through three discourse-level aspects. We show that explicit integration of discourse features can enhance storytelling, as is demonstrated by over 40% improvement in neural storytelling.
  arXiv  Detail & Related papers  (2024-07-18T08:02:49Z)
• NarraSum: A Large-Scale Dataset for Abstractive Narrative Summarization [26.80378373420446]
  NarraSum is a large-scale narrative summarization dataset. It contains 122K narrative documents, which are collected from plot descriptions of movies and TV episodes with diverse genres, and their corresponding abstractive summaries. Experiments show that there is a large performance gap between humans and the state-of-the-art summarization models on NarraSum.
  arXiv  Detail & Related papers  (2022-12-02T22:51:51Z)
• Paragraph-level Commonsense Transformers with Recurrent Memory [77.4133779538797]
  We train a discourse-aware model that incorporates paragraph-level information to generate coherent commonsense inferences from narratives. Our results show that PARA-COMET outperforms the sentence-level baselines, particularly in generating inferences that are both coherent and novel.
  arXiv  Detail & Related papers  (2020-10-04T05:24:12Z)
• Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
  We present the first application of the BERTSum model to conversational language. We generate abstractive summaries of narrated instructional videos across a wide variety of topics. We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
  arXiv  Detail & Related papers  (2020-08-21T20:59:34Z)
• CompRes: A Dataset for Narrative Structure in News [2.4578723416255754]
  We introduce CompRes -- the first dataset for narrative structure in news media. We use the annotated dataset to train several supervised models to identify the different narrative elements.
  arXiv  Detail & Related papers  (2020-07-09T15:21:59Z)
• PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking [128.76063992147016]
  We present PlotMachines, a neural narrative model that learns to transform an outline into a coherent story by tracking the dynamic plot states. In addition, we enrich PlotMachines with high-level discourse structure so that the model can learn different writing styles corresponding to different parts of the narrative.
  arXiv  Detail & Related papers  (2020-04-30T17:16:31Z)
• Screenplay Summarization Using Latent Narrative Structure [78.45316339164133]
  We propose to explicitly incorporate the underlying structure of narratives into general unsupervised and supervised extractive summarization models. We formalize narrative structure in terms of key narrative events (turning points) and treat it as latent in order to summarize screenplays. Experimental results on the CSI corpus of TV screenplays, which we augment with scene-level summarization labels, show that latent turning points correlate with important aspects of a CSI episode.
  arXiv  Detail & Related papers  (2020-04-27T11:54:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.