Exploration of Plan-Guided Summarization for Narrative Texts: the Case of Small Language Models

• URL: http://arxiv.org/abs/2504.09071v1
• Date: Sat, 12 Apr 2025 04:11:37 GMT
• Title: Exploration of Plan-Guided Summarization for Narrative Texts: the Case of Small Language Models
• Authors: Matt Grenander, Siddharth Varia, Paula Czarnowska, Yogarshi Vyas, Kishaloy Halder, Bonan Min,
• Abstract summary: Plan-guided summarization attempts to reduce hallucinations in small language models (SLMs) by grounding generated summaries to the source text.<n>We analyze existing plan-guided solutions targeting fine-grained details, and also propose our own higher-level, narrative-based plan formulation.<n>Our results show that neither approach significantly improves on a baseline without planning in either summary quality or faithfulness.
• Score: 15.056004370209916
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Plan-guided summarization attempts to reduce hallucinations in small language models (SLMs) by grounding generated summaries to the source text, typically by targeting fine-grained details such as dates or named entities. In this work, we investigate whether plan-based approaches in SLMs improve summarization in long document, narrative tasks. Narrative texts' length and complexity often mean they are difficult to summarize faithfully. We analyze existing plan-guided solutions targeting fine-grained details, and also propose our own higher-level, narrative-based plan formulation. Our results show that neither approach significantly improves on a baseline without planning in either summary quality or faithfulness. Human evaluation reveals that while plan-guided approaches are often well grounded to their plan, plans are equally likely to contain hallucinations compared to summaries. As a result, the plan-guided summaries are just as unfaithful as those from models without planning. Our work serves as a cautionary tale to plan-guided approaches to summarization, especially for long, complex domains such as narrative texts.

Related papers

• Explanatory Summarization with Discourse-Driven Planning [58.449423507036414]
  We present a plan-based approach that leverages discourse frameworks to organize summary generation and guide explanatory sentences. Specifically, we propose two discourse-driven planning strategies, where the plan is conditioned as part of the input or part of the output prefix. Empirical experiments on three lay summarization datasets show that our approach outperforms existing state-of-the-art methods in terms of summary quality.
  arXiv  Detail & Related papers  (2025-04-27T19:47:36Z)
• Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks [36.63527489464188]
  Plan-and-Act is a framework that incorporates explicit planning into large language models (LLMs)<n>Plan-and-Act consists of a Planner model which generates structured, high-level plans to achieve user goals, and an Executor model that translates these plans into environment-specific actions.<n>We evaluate Plan-and-Act using web navigation as a representative long-horizon planning environment, demonstrating a state-of-the-art 54% success rate on the WebArena-Lite benchmark.
  arXiv  Detail & Related papers  (2025-03-12T17:40:52Z)
• Analysis of Plan-based Retrieval for Grounded Text Generation [78.89478272104739]
  hallucinations occur when a language model is given a generation task outside its parametric knowledge. A common strategy to address this limitation is to infuse the language models with retrieval mechanisms. We analyze how planning can be used to guide retrieval to further reduce the frequency of hallucinations.
  arXiv  Detail & Related papers  (2024-08-20T02:19:35Z)
• PLANTS: A Novel Problem and Dataset for Summarization of Planning-Like (PL) Tasks [6.408228255254908]
  We introduce a novel plan summarization problem, presenting a dataset, and providing a baseline method for generating PL summaries. We believe the novel problem and dataset can reinvigorate research in summarization, which some consider as a solved problem.
  arXiv  Detail & Related papers  (2024-07-18T15:36:02Z)
• Exploring and Benchmarking the Planning Capabilities of Large Language Models [57.23454975238014]
  This work lays the foundations for improving planning capabilities of large language models (LLMs) We construct a comprehensive benchmark suite encompassing both classical planning benchmarks and natural language scenarios. We investigate the use of many-shot in-context learning to enhance LLM planning, exploring the relationship between increased context length and improved planning performance.
  arXiv  Detail & Related papers  (2024-06-18T22:57:06Z)
• EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form Narrative Text Generation [114.50719922069261]
  We propose a new framework called Evaluation-guided Iterative Plan Extraction for long-form narrative text generation (EIPE-text) EIPE-text has three stages: plan extraction, learning, and inference. We evaluate the effectiveness of EIPE-text in the domains of novels and storytelling.
  arXiv  Detail & Related papers  (2023-10-12T10:21:37Z)
• Zero-Shot On-the-Fly Event Schema Induction [61.91468909200566]
  We present a new approach in which large language models are utilized to generate source documents that allow predicting, given a high-level event definition, the specific events, arguments, and relations between them. Using our model, complete schemas on any topic can be generated on-the-fly without any manual data collection, i.e., in a zero-shot manner.
  arXiv  Detail & Related papers  (2022-10-12T14:37:00Z)
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries [46.183289748907804]
  We propose SOE, a pipelined system that outlines, outlining and elaborating for long text generation. SOE produces long texts with significantly better quality, along with faster convergence speed.
  arXiv  Detail & Related papers  (2020-10-14T13:22:20Z)
• Screenplay Summarization Using Latent Narrative Structure [78.45316339164133]
  We propose to explicitly incorporate the underlying structure of narratives into general unsupervised and supervised extractive summarization models. We formalize narrative structure in terms of key narrative events (turning points) and treat it as latent in order to summarize screenplays. Experimental results on the CSI corpus of TV screenplays, which we augment with scene-level summarization labels, show that latent turning points correlate with important aspects of a CSI episode.
  arXiv  Detail & Related papers  (2020-04-27T11:54:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.