PORTRAIT: a hybrid aPproach tO cReate extractive ground-TRuth summAry for dIsaster evenT

• URL: http://arxiv.org/abs/2305.11536v1
• Date: Fri, 19 May 2023 09:07:52 GMT
• Title: PORTRAIT: a hybrid aPproach tO cReate extractive ground-TRuth summAry for dIsaster evenT
• Authors: Piyush Kumar Garg, Roshni Chakraborty, and Sourav Kumar Dandapat
• Abstract summary: Disaster summarization approaches provide an overview of the important information posted during disaster events on social media platforms, such as, Twitter. We propose a hybrid (semi-automated) approach (PORTRAIT) where we partly automate the ground-truth summary generation procedure. We validate the effectiveness of PORTRAIT on 5 disaster events through quantitative and qualitative comparisons of ground-truth summaries generated by existing intuitive approaches, a semi-automated approach, and PORTRAIT.
• Score: 5.386050544766801
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Disaster summarization approaches provide an overview of the important information posted during disaster events on social media platforms, such as, Twitter. However, the type of information posted significantly varies across disasters depending on several factors like the location, type, severity, etc. Verification of the effectiveness of disaster summarization approaches still suffer due to the lack of availability of good spectrum of datasets along with the ground-truth summary. Existing approaches for ground-truth summary generation (ground-truth for extractive summarization) relies on the wisdom and intuition of the annotators. Annotators are provided with a complete set of input tweets from which a subset of tweets is selected by the annotators for the summary. This process requires immense human effort and significant time. Additionally, this intuition-based selection of the tweets might lead to a high variance in summaries generated across annotators. Therefore, to handle these challenges, we propose a hybrid (semi-automated) approach (PORTRAIT) where we partly automate the ground-truth summary generation procedure. This approach reduces the effort and time of the annotators while ensuring the quality of the created ground-truth summary. We validate the effectiveness of PORTRAIT on 5 disaster events through quantitative and qualitative comparisons of ground-truth summaries generated by existing intuitive approaches, a semi-automated approach, and PORTRAIT. We prepare and release the ground-truth summaries for 5 disaster events which consist of both natural and man-made disaster events belonging to 4 different countries. Finally, we provide a study about the performance of various state-of-the-art summarization approaches on the ground-truth summaries generated by PORTRAIT using ROUGE-N F1-scores.

Related papers

• ADSumm: Annotated Ground-truth Summary Datasets for Disaster Tweet Summarization [8.371475703337106]
  Existing tweet disaster summarization approaches provide a summary of these events to aid government agencies, humanitarian organizations, etc. In this paper, we present ADSumm, which adds annotated ground-truth summaries for eight disaster events. Our experimental analysis shows that the newly added datasets improve the performance of the supervised summarization approaches by 8-28% in terms of ROUGE-N F1-score.
  arXiv  Detail & Related papers  (2024-05-10T15:49:01Z)
• AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning [98.26836657967162]
  textbfAgentOhana aggregates agent trajectories from distinct environments, spanning a wide array of scenarios. textbfxLAM-v0.1, a large action model tailored for AI agents, demonstrates exceptional performance across various benchmarks.
  arXiv  Detail & Related papers  (2024-02-23T18:56:26Z)
• Multi-Query Focused Disaster Summarization via Instruction-Based Prompting [3.6199702611839792]
  CrisisFACTS aims to advance disaster summarization based on multi-stream fact-finding. Here, participants are asked to develop systems that can extract key facts from several disaster-related events. This paper describes our method to tackle this challenging task.
  arXiv  Detail & Related papers  (2024-02-14T08:22:58Z)
• AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation [57.8363998797433]
  We propose AMRFact, a framework that generates perturbed summaries using Abstract Meaning Representations (AMRs) Our approach parses factually consistent summaries into AMR graphs and injects controlled factual inconsistencies to create negative examples, allowing for coherent factually inconsistent summaries to be generated with high error-type coverage.
  arXiv  Detail & Related papers  (2023-11-16T02:56:29Z)
• CrisisMatch: Semi-Supervised Few-Shot Learning for Fine-Grained Disaster Tweet Classification [51.58605842457186]
  We present a fine-grained disaster tweet classification model under the semi-supervised, few-shot learning setting. Our model, CrisisMatch, effectively classifies tweets into fine-grained classes of interest using few labeled data and large amounts of unlabeled data.
  arXiv  Detail & Related papers  (2023-10-23T07:01:09Z)
• Benchmarking the Generation of Fact Checking Explanations [19.363672064425504]
  We focus on the generation of justifications (textual explanation of why a claim is classified as either true or false) and benchmark it with novel datasets and advanced baselines. Results show that in justification production summarization benefits from the claim information. Although cross-dataset experiments suffer from performance degradation, a unique model trained on a combination of the two datasets is able to retain style information in an efficient manner.
  arXiv  Detail & Related papers  (2023-08-29T10:40:46Z)
• IKDSumm: Incorporating Key-phrases into BERT for extractive Disaster Tweet Summarization [5.299958874647294]
  We propose a disaster-specific tweet summarization framework, IKDSumm. IKDSumm identifies the crucial and important information from each tweet related to a disaster through key-phrases of that tweet. We utilize these key-phrases to automatically generate a summary of the tweets.
  arXiv  Detail & Related papers  (2023-05-19T11:05:55Z)
• Boosting Event Extraction with Denoised Structure-to-Text Augmentation [52.21703002404442]
  Event extraction aims to recognize pre-defined event triggers and arguments from texts. Recent data augmentation methods often neglect the problem of grammatical incorrectness. We propose a denoised structure-to-text augmentation framework for event extraction DAEE.
  arXiv  Detail & Related papers  (2023-05-16T16:52:07Z)
• Salience Allocation as Guidance for Abstractive Summarization [61.31826412150143]
  We propose a novel summarization approach with a flexible and reliable salience guidance, namely SEASON (SaliencE Allocation as Guidance for Abstractive SummarizatiON) SEASON utilizes the allocation of salience expectation to guide abstractive summarization and adapts well to articles in different abstractiveness.
  arXiv  Detail & Related papers  (2022-10-22T02:13:44Z)
• Multi-Fact Correction in Abstractive Text Summarization [98.27031108197944]
  Span-Fact is a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-10-06T02:51:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.