Event-based evaluation of abstractive news summarization

• URL: http://arxiv.org/abs/2507.01160v1
• Date: Tue, 01 Jul 2025 19:49:23 GMT
• Title: Event-based evaluation of abstractive news summarization
• Authors: Huiling You, Samia Touileb, Erik Velldal, Lilja Øvrelid,
• Abstract summary: We evaluate the quality of abstractive summaries by calculating overlapping events between generated summaries, reference summaries, and the original news articles.<n>Our approach provides more insight into the event information contained in the summaries.
• Score: 8.25219440625445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An abstractive summary of a news article contains its most important information in a condensed version. The evaluation of automatically generated summaries by generative language models relies heavily on human-authored summaries as gold references, by calculating overlapping units or similarity scores. News articles report events, and ideally so should the summaries. In this work, we propose to evaluate the quality of abstractive summaries by calculating overlapping events between generated summaries, reference summaries, and the original news articles. We experiment on a richly annotated Norwegian dataset comprising both events annotations and summaries authored by expert human annotators. Our approach provides more insight into the event information contained in the summaries.

Related papers

• Consistency Evaluation of News Article Summaries Generated by Large (and Small) Language Models [0.0]
  Large Language Models (LLMs) have shown promise in generating fluent abstractive summaries but they can produce hallucinated details not grounded in the source text.<n>This paper embarks on an exploration of text summarization with a diverse set of techniques, including TextRank, BART, Mistral-7B-Instruct, and OpenAI GPT-3.5-Turbo.<n>We find that all summarization models produce consistent summaries when tested on the XL-Sum dataset.
  arXiv  Detail & Related papers  (2025-02-28T01:58:17Z)
• AugSumm: towards generalizable speech summarization using synthetic labels from large language model [61.73741195292997]
  Abstractive speech summarization (SSUM) aims to generate human-like summaries from speech. conventional SSUM models are mostly trained and evaluated with a single ground-truth (GT) human-annotated deterministic summary. We propose AugSumm, a method to leverage large language models (LLMs) as a proxy for human annotators to generate augmented summaries.
  arXiv  Detail & Related papers  (2024-01-10T18:39:46Z)
• Automatic News Summerization [0.0]
  The study employs the CNN-Daily Mail dataset, which consists of news articles and human-generated reference summaries. The evaluation employs ROUGE scores to assess the efficacy and quality of generated summaries.
  arXiv  Detail & Related papers  (2023-10-17T18:38:03Z)
• SummIt: Iterative Text Summarization via ChatGPT [12.966825834765814]
  We propose SummIt, an iterative text summarization framework based on large language models like ChatGPT. Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback. We also conduct a human evaluation to validate the effectiveness of the iterative refinements and identify a potential issue of over-correction.
  arXiv  Detail & Related papers  (2023-05-24T07:40:06Z)
• Salience Allocation as Guidance for Abstractive Summarization [61.31826412150143]
  We propose a novel summarization approach with a flexible and reliable salience guidance, namely SEASON (SaliencE Allocation as Guidance for Abstractive SummarizatiON) SEASON utilizes the allocation of salience expectation to guide abstractive summarization and adapts well to articles in different abstractiveness.
  arXiv  Detail & Related papers  (2022-10-22T02:13:44Z)
• Podcast Summary Assessment: A Resource for Evaluating Summary Assessment Methods [42.08097583183816]
  We describe a new dataset, the podcast summary assessment corpus. This dataset has two unique aspects: (i) long-input, speech podcast based, documents; and (ii) an opportunity to detect inappropriate reference summaries in podcast corpus.
  arXiv  Detail & Related papers  (2022-08-28T18:24:41Z)
• Template-based Abstractive Microblog Opinion Summarisation [26.777997436856076]
  We introduce the task of microblog opinion summarisation (MOS) and share a dataset of 3100 gold-standard opinion summaries. The dataset contains summaries of tweets spanning a 2-year period and covers more topics than any other public Twitter summarisation dataset.
  arXiv  Detail & Related papers  (2022-08-08T12:16:01Z)
• Unsupervised Summarization with Customized Granularities [76.26899748972423]
  We propose the first unsupervised multi-granularity summarization framework, GranuSum. By inputting different numbers of events, GranuSum is capable of producing multi-granular summaries in an unsupervised manner.
  arXiv  Detail & Related papers  (2022-01-29T05:56:35Z)
• Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning [66.30909748400023]
  We propose to evaluate the summary qualities without reference summaries by unsupervised contrastive learning. Specifically, we design a new metric which covers both linguistic qualities and semantic informativeness based on BERT. Experiments on Newsroom and CNN/Daily Mail demonstrate that our new evaluation method outperforms other metrics even without reference summaries.
  arXiv  Detail & Related papers  (2020-10-05T05:04:14Z)
• Few-Shot Learning for Opinion Summarization [117.70510762845338]
  Opinion summarization is the automatic creation of text reflecting subjective information expressed in multiple documents. In this work, we show that even a handful of summaries is sufficient to bootstrap generation of the summary text. Our approach substantially outperforms previous extractive and abstractive methods in automatic and human evaluation.
  arXiv  Detail & Related papers  (2020-04-30T15:37:38Z)
• Screenplay Summarization Using Latent Narrative Structure [78.45316339164133]
  We propose to explicitly incorporate the underlying structure of narratives into general unsupervised and supervised extractive summarization models. We formalize narrative structure in terms of key narrative events (turning points) and treat it as latent in order to summarize screenplays. Experimental results on the CSI corpus of TV screenplays, which we augment with scene-level summarization labels, show that latent turning points correlate with important aspects of a CSI episode.
  arXiv  Detail & Related papers  (2020-04-27T11:54:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.