ROUGE-K: Do Your Summaries Have Keywords?

• URL: http://arxiv.org/abs/2403.05186v1
• Date: Fri, 8 Mar 2024 09:54:56 GMT
• Title: ROUGE-K: Do Your Summaries Have Keywords?
• Authors: Sotaro Takeshita, Simone Paolo Ponzetto, Kai Eckert
• Abstract summary: Keywords, that is, content-relevant words in summaries play an important role in efficient information conveyance. Existing evaluation metrics for extreme summarization models do not pay explicit attention to keywords in summaries. We propose four approaches for incorporating word importance into a transformer-based model.
• Score: 11.393728547335217
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Keywords, that is, content-relevant words in summaries play an important role in efficient information conveyance, making it critical to assess if system-generated summaries contain such informative words during evaluation. However, existing evaluation metrics for extreme summarization models do not pay explicit attention to keywords in summaries, leaving developers ignorant of their presence. To address this issue, we present a keyword-oriented evaluation metric, dubbed ROUGE-K, which provides a quantitative answer to the question of -- \textit{How well do summaries include keywords?} Through the lens of this keyword-aware metric, we surprisingly find that a current strong baseline model often misses essential information in their summaries. Our analysis reveals that human annotators indeed find the summaries with more keywords to be more relevant to the source documents. This is an important yet previously overlooked aspect in evaluating summarization systems. Finally, to enhance keyword inclusion, we propose four approaches for incorporating word importance into a transformer-based model and experimentally show that it enables guiding models to include more keywords while keeping the overall quality. Our code is released at https://github.com/sobamchan/rougek.

Related papers

• Hierarchical Indexing for Retrieval-Augmented Opinion Summarization [60.5923941324953]
  We propose a method for unsupervised abstractive opinion summarization that combines the attributability and scalability of extractive approaches with the coherence and fluency of Large Language Models (LLMs) Our method, HIRO, learns an index structure that maps sentences to a path through a semantically organized discrete hierarchy. At inference time, we populate the index and use it to identify and retrieve clusters of sentences containing popular opinions from input reviews.
  arXiv  Detail & Related papers  (2024-03-01T10:38:07Z)
• Quantifying the redundancy between prosody and text [67.07817268372743]
  We use large language models to estimate how much information is redundant between prosody and the words themselves. We find a high degree of redundancy between the information carried by the words and prosodic information across several prosodic features. Still, we observe that prosodic features can not be fully predicted from text, suggesting that prosody carries information above and beyond the words.
  arXiv  Detail & Related papers  (2023-11-28T21:15:24Z)
• On Context Utilization in Summarization with Large Language Models [83.84459732796302]
  Large language models (LLMs) excel in abstractive summarization tasks, delivering fluent and pertinent summaries. Recent advancements have extended their capabilities to handle long-input contexts, exceeding 100k tokens. We conduct the first comprehensive study on context utilization and position bias in summarization.
  arXiv  Detail & Related papers  (2023-10-16T16:45:12Z)
• KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation [69.57018875757622]
  We propose KPEval, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. Using KPEval, we re-evaluate 23 keyphrase systems and discover that established model comparison results have blind-spots.
  arXiv  Detail & Related papers  (2023-03-27T17:45:38Z)
• Applying Transformer-based Text Summarization for Keyphrase Generation [2.28438857884398]
  Keyphrases are crucial for searching and systematizing scholarly documents. In this paper, we experiment with popular transformer-based models for abstractive text summarization. We show that summarization models are quite effective in generating keyphrases in the terms of the full-match F1-score and BERT.Score. We also investigate several ordering strategies to target keyphrases.
  arXiv  Detail & Related papers  (2022-09-08T13:01:52Z)
• Keyphrase Extraction Using Neighborhood Knowledge Based on Word Embeddings [17.198907789163123]
  We enhance the graph-based ranking model by leveraging word embeddings as background knowledge to add semantic information to the inter-word graph. Our approach is evaluated on established benchmark datasets and empirical results show that the word embedding neighborhood information improves the model performance.
  arXiv  Detail & Related papers  (2021-11-13T21:48:18Z)
• Deep Keyphrase Completion [59.0413813332449]
  Keyphrase provides accurate information of document content that is highly compact, concise, full of meanings, and widely used for discourse comprehension, organization, and text retrieval. We propose textitkeyphrase completion (KPC) to generate more keyphrases for document (e.g. scientific publication) taking advantage of document content along with a very limited number of known keyphrases. We name it textitdeep keyphrase completion (DKPC) since it attempts to capture the deep semantic meaning of the document content together with known keyphrases via a deep learning framework
  arXiv  Detail & Related papers  (2021-10-29T07:15:35Z)
• Importance Estimation from Multiple Perspectives for Keyphrase Extraction [34.51718374923614]
  We propose a new approach to estimate the importance of keyphrase from multiple perspectives (called as textitKIEMP) textitKIEMP estimates the importance of phrase with three modules: a chunking module to measure its syntactic accuracy, a ranking module to check its information saliency, and a matching module to judge the concept consistency between phrase and the whole document. Experimental results on six benchmark datasets show that textitKIEMP outperforms the existing state-of-the-art keyphrase extraction approaches in most cases.
  arXiv  Detail & Related papers  (2021-10-19T05:48:22Z)
• FRAKE: Fusional Real-time Automatic Keyword Extraction [1.332091725929965]
  Keywords extraction is called identifying words or phrases that express the main concepts of texts in best. We use a combined approach that consists of two models of graph centrality features and textural features.
  arXiv  Detail & Related papers  (2021-04-10T18:30:17Z)
• Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoders [59.038157066874255]
  We propose a novel framework called RankAE to perform chat summarization without employing manually labeled data. RankAE consists of a topic-oriented ranking strategy that selects topic utterances according to centrality and diversity simultaneously. A denoising auto-encoder is designed to generate succinct but context-informative summaries based on the selected utterances.
  arXiv  Detail & Related papers  (2020-12-14T07:31:17Z)
• Extending Text Informativeness Measures to Passage Interestingness Evaluation (Language Model vs. Word Embedding) [1.2998637003026272]
  This paper defines the concept of Interestingness as a generalization of Informativeness. We then study the ability of state of the art Informativeness measures to cope with this generalization. We prove that the CLEF-INEX Tweet Contextualization 2012 Logarithm Similarity measure provides best results.
  arXiv  Detail & Related papers  (2020-04-14T18:22:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.