Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries

• URL: http://arxiv.org/abs/2604.06416v1
• Date: Tue, 07 Apr 2026 19:50:52 GMT
• Title: Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries
• Abstract summary: We evaluate whether models mirror human patterns of conceptual engagement with texts.<n>We align sentences from 150 human-written novel summaries with the specific chapters they reference.<n>We find both stylistic differences between the texts and differences in how humans and LLMs distribute their focus throughout a narrative.
• Score: 4.320927217095906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although LLM context lengths have grown, there is evidence that their ability to integrate information across long-form texts has not kept pace. We evaluate one such understanding task: generating summaries of novels. When human authors of summaries compress a story, they reveal what they consider narratively important. Therefore, by comparing human and LLM-authored summaries, we can assess whether models mirror human patterns of conceptual engagement with texts. To measure conceptual engagement, we align sentences from 150 human-written novel summaries with the specific chapters they reference. We demonstrate the difficulty of this alignment task, which indicates the complexity of summarization as a task. We then generate and align additional summaries by nine state-of-the-art LLMs for each of the 150 reference texts. Comparing the human and model-authored summaries, we find both stylistic differences between the texts and differences in how humans and LLMs distribute their focus throughout a narrative, with models emphasizing the ends of texts. Comparing human narrative engagement with model attention mechanisms suggests explanations for degraded narrative comprehension and targets for future development. We release our dataset to support future research.

Related papers

• The Plot Thins: Uniformity and Linearity in Literary Summaries [4.320927217095906]
  We construct a dataset mapping sentences from 150 novel summaries to their respective source chapters.<n>We measure summary linearity, the degree to which it maintains the source's order of events, and uniformity, the degree to which a summary spreads attention equally across a source.
  arXiv  Detail & Related papers  (2026-08-18T00:10:39Z)
• Do Large Language Models Always Tell The Same Stories? [2.4779770815686444]
  We investigate the diversity of large language models-generated stories through the framework of narrative similarity.<n>Our findings reveal a consistent trend: LLM-generated narratives are consistently more similar to each other than human-written stories are.<n>We show that common mitigation strategies, including negative prompting and temperature scaling, fail to meaningfully address this homogeneity.
  arXiv  Detail & Related papers  (2026-06-15T22:52:02Z)
• Creative Convergence or Imitation? Genre-Specific Homogeneity in LLM-Generated Chinese Literature [17.985893852336865]
  We propose a novel theoretical framework for analysis by incorporating Proppian narratology and narrative functions.<n>We extend Propp's narrative theory, defining 34 narrative functions suited to modern web narrative structures.<n> Experiments reveal that the primary reasons for the singular narrative logic and severe homogenization in generated texts are that current LLMs are unable to correctly comprehend the meanings of narrative functions.
  arXiv  Detail & Related papers  (2026-03-15T15:15:48Z)
• EvolvTrip: Enhancing Literary Character Understanding with Temporal Theory-of-Mind Graphs [23.86303464364475]
  We introduce EvolvTrip, a perspective-aware temporal knowledge graph that tracks psychological development throughout narratives.<n>Our findings highlight the importance of explicit representation of temporal character mental states in narrative comprehension.
  arXiv  Detail & Related papers  (2025-06-16T16:05:17Z)
• The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters [67.61587661660852]
  Theory-of-Mind (ToM) allows humans to understand and interpret the mental states of others.<n>In this paper, we verify the importance of comprehensive contextual understanding about personal backgrounds in ToM.<n>We introduce CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels.
  arXiv  Detail & Related papers  (2025-01-03T09:04:45Z)
• BookWorm: A Dataset for Character Description and Analysis [59.186325346763184]
  We define two tasks: character description, which generates a brief factual profile, and character analysis, which offers an in-depth interpretation. We introduce the BookWorm dataset, pairing books from the Gutenberg Project with human-written descriptions and analyses. Our findings show that retrieval-based approaches outperform hierarchical ones in both tasks.
  arXiv  Detail & Related papers  (2024-10-14T10:55:58Z)
• A Modular Approach for Multimodal Summarization of TV Shows [55.20132267309382]
  We present a modular approach where separate components perform specialized sub-tasks. Our modules involve detecting scene boundaries, reordering scenes so as to minimize the number of cuts between different events, converting visual information to text, summarizing the dialogue in each scene, and fusing the scene summaries into a final summary for the entire episode. We also present a new metric, PRISMA, to measure both precision and recall of generated summaries, which we decompose into atomic facts.
  arXiv  Detail & Related papers  (2024-03-06T16:10:01Z)
• Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers [25.268709339109893]
  We evaluate recent Large Language Models (LLMs) on the challenging task of summarizing short stories. We work directly with authors to ensure that the stories have not been shared online (and therefore are unseen by the models) We compare GPT-4, Claude-2.1, and LLama-2-70B and find that all three models make faithfulness mistakes in over 50% of summaries.
  arXiv  Detail & Related papers  (2024-03-02T01:52:14Z)
• Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding [21.900015612952146]
  Narrative understanding involves capturing the author's cognitive processes, providing insights into their knowledge, intentions, beliefs, and desires. Although large language models (LLMs) excel in generating grammatically coherent text, their ability to comprehend the author's thoughts remains uncertain. This hinders the practical applications of narrative understanding.
  arXiv  Detail & Related papers  (2023-10-28T18:47:57Z)
• Summarization is (Almost) Dead [49.360752383801305]
  We develop new datasets and conduct human evaluation experiments to evaluate the zero-shot generation capability of large language models (LLMs) Our findings indicate a clear preference among human evaluators for LLM-generated summaries over human-written summaries and summaries generated by fine-tuned models.
  arXiv  Detail & Related papers  (2023-09-18T08:13:01Z)
• SNaC: Coherence Error Detection for Narrative Summarization [73.48220043216087]
  We introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie screenplay summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowd annotators.
  arXiv  Detail & Related papers  (2022-05-19T16:01:47Z)
• Summarization with Graphical Elements [55.5913491389047]
  We propose a new task: summarization with graphical elements. We collect a high quality human labeled dataset to support research into the task.
  arXiv  Detail & Related papers  (2022-04-15T17:16:41Z)
• The Shmoop Corpus: A Dataset of Stories with Loosely Aligned Summaries [72.48439126769627]
  We introduce the Shmoop Corpus: a dataset of 231 stories paired with detailed multi-paragraph summaries for each individual chapter. From the corpus, we construct a set of common NLP tasks, including Cloze-form question answering and a simplified form of abstractive summarization. We believe that the unique structure of this corpus provides an important foothold towards making machine story comprehension more approachable.
  arXiv  Detail & Related papers  (2019-12-30T21:03:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.