GUMsley: Evaluating Entity Salience in Summarization for 12 English Genres

• URL: http://arxiv.org/abs/2401.17974v1
• Date: Wed, 31 Jan 2024 16:30:50 GMT
• Title: GUMsley: Evaluating Entity Salience in Summarization for 12 English Genres
• Authors: Jessica Lin, Amir Zeldes
• Abstract summary: We present and evaluate GUMsley, the first entity salience dataset covering all named and non-named salient entities for 12 genres of English text. We show that predicting or providing salient entities to several model architectures enhances performance and helps derive higher-quality summaries.
• Score: 14.37990666928991
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As NLP models become increasingly capable of understanding documents in terms of coherent entities rather than strings, obtaining the most salient entities for each document is not only an important end task in itself but also vital for Information Retrieval (IR) and other downstream applications such as controllable summarization. In this paper, we present and evaluate GUMsley, the first entity salience dataset covering all named and non-named salient entities for 12 genres of English text, aligned with entity types, Wikification links and full coreference resolution annotations. We promote a strict definition of salience using human summaries and demonstrate high inter-annotator agreement for salience based on whether a source entity is mentioned in the summary. Our evaluation shows poor performance by pre-trained SOTA summarization models and zero-shot LLM prompting in capturing salient entities in generated summaries. We also show that predicting or providing salient entities to several model architectures enhances performance and helps derive higher-quality summaries by alleviating the entity hallucination problem in existing abstractive summarization.

Related papers

• Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs [70.15262704746378]
  We propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback. Preliminary experiments with Falcon-40B and Llama-2-13B show significant performance improvements (10% Rouge-L) in terms of producing coherent summaries.
  arXiv  Detail & Related papers  (2024-07-05T20:25:04Z)
• Entity Disambiguation via Fusion Entity Decoding [68.77265315142296]
  We propose an encoder-decoder model to disambiguate entities with more detailed entity descriptions. We observe +1.5% improvements in end-to-end entity linking in the GERBIL benchmark compared with EntQA.
  arXiv  Detail & Related papers  (2024-04-02T04:27:54Z)
• On Context Utilization in Summarization with Large Language Models [83.84459732796302]
  Large language models (LLMs) excel in abstractive summarization tasks, delivering fluent and pertinent summaries. Recent advancements have extended their capabilities to handle long-input contexts, exceeding 100k tokens. We conduct the first comprehensive study on context utilization and position bias in summarization.
  arXiv  Detail & Related papers  (2023-10-16T16:45:12Z)
• GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization [10.609715843964263]
  Automatic summarization with pre-trained language models has led to impressively fluent results, but is prone to 'hallucinations' We present GUMSum, a dataset of English summaries in 12 written and spoken genres for evaluation of abstractive summarization.
  arXiv  Detail & Related papers  (2023-06-20T03:21:10Z)
• LMGQS: A Large-scale Dataset for Query-focused Summarization [77.6179359525065]
  We convert four generic summarization benchmarks into a new QFS benchmark dataset, LMGQS. We establish baselines with state-of-the-art summarization models. We achieve state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks.
  arXiv  Detail & Related papers  (2023-05-22T14:53:45Z)
• Unsupervised Summarization with Customized Granularities [76.26899748972423]
  We propose the first unsupervised multi-granularity summarization framework, GranuSum. By inputting different numbers of events, GranuSum is capable of producing multi-granular summaries in an unsupervised manner.
  arXiv  Detail & Related papers  (2022-01-29T05:56:35Z)
• AgreeSum: Agreement-Oriented Multi-Document Summarization [3.4743618614284113]
  Given a cluster of articles, the goal is to provide abstractive summaries that represent information common and faithful to all input articles. We create a dataset for AgreeSum, and provide annotations on articlesummary entailment relations for a subset of the clusters in the dataset.
  arXiv  Detail & Related papers  (2021-06-04T06:17:49Z)
• Entity-level Factual Consistency of Abstractive Text Summarization [26.19686599842915]
  Key challenge for abstractive summarization is ensuring factual consistency of the generated summary with respect to the original document. We propose a set of new metrics to quantify the entity-level factual consistency of generated summaries.
  arXiv  Detail & Related papers  (2021-02-18T03:07:28Z)
• Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation [93.87095877617968]
  We propose Constrained Abstractive Summarization (CAS), a general setup that preserves the factual consistency of abstractive summarization. We adopt lexically constrained decoding, a technique generally applicable to autoregressive generative models, to fulfill CAS. We observe up to 13.8 ROUGE-2 gains when only one manual constraint is used in interactive summarization.
  arXiv  Detail & Related papers  (2020-10-24T00:27:44Z)
• SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling [25.299937353444854]
  We present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries. Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries.
  arXiv  Detail & Related papers  (2020-05-13T15:40:13Z)
• Neural Entity Summarization with Joint Encoding and Weak Supervision [29.26714907483851]
  In knowledge graphs, an entity is often described by a large number of triple facts. Existing solutions to entitymarization are mainly unsupervised. We present a supervised approach that is based on our novel neural model.
  arXiv  Detail & Related papers  (2020-05-01T00:14:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.