MAVEN-Fact: A Large-scale Event Factuality Detection Dataset

• URL: http://arxiv.org/abs/2407.15352v1
• Date: Mon, 22 Jul 2024 03:43:46 GMT
• Title: MAVEN-Fact: A Large-scale Event Factuality Detection Dataset
• Authors: Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu, Juanzi Li,
• Abstract summary: We introduce MAVEN-Fact, a large-scale and high-quality EFD dataset based on the MAVEN dataset. MAVEN-Fact includes factuality annotations of 112,276 events, making it the largest EFD dataset. Experiments demonstrate that MAVEN-Fact is challenging for both conventional fine-tuned models and large language models (LLMs)
• Score: 55.01875707021496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Event Factuality Detection (EFD) task determines the factuality of textual events, i.e., classifying whether an event is a fact, possibility, or impossibility, which is essential for faithfully understanding and utilizing event knowledge. However, due to the lack of high-quality large-scale data, event factuality detection is under-explored in event understanding research, which limits the development of EFD community. To address these issues and provide faithful event understanding, we introduce MAVEN-Fact, a large-scale and high-quality EFD dataset based on the MAVEN dataset. MAVEN-Fact includes factuality annotations of 112,276 events, making it the largest EFD dataset. Extensive experiments demonstrate that MAVEN-Fact is challenging for both conventional fine-tuned models and large language models (LLMs). Thanks to the comprehensive annotations of event arguments and relations in MAVEN, MAVEN-Fact also supports some further analyses and we find that adopting event arguments and relations helps in event factuality detection for fine-tuned models but does not benefit LLMs. Furthermore, we preliminarily study an application case of event factuality detection and find it helps in mitigating event-related hallucination in LLMs. Our dataset and codes can be obtained from \url{https://github.com/lcy2723/MAVEN-FACT}

Related papers

• Enhancing Event Reasoning in Large Language Models through Instruction Fine-Tuning with Semantic Causal Graphs [7.9482258228279195]
  We propose a novel approach for instruction fine-tuning LLMs for event detection. Our method introduces Semantic Causal Graphs (SCGs) to capture both causal relationships and contextual information within text. Our evaluations demonstrate that training LLMs with SCG Instructions outperforms standard instruction fine-tuning by an average of 35.69% on Event Trigger Classification.
  arXiv  Detail & Related papers  (2024-08-30T18:56:06Z)
• Improving Event Definition Following For Zero-Shot Event Detection [66.27883872707523]
  Existing approaches on zero-shot event detection usually train models on datasets annotated with known event types. We aim to improve zero-shot event detection by training models to better follow event definitions.
  arXiv  Detail & Related papers  (2024-03-05T01:46:50Z)
• MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation [104.6065882758648]
  MAVEN-Arg is the first all-in-one dataset supporting event detection, event argument extraction, and event relation extraction. As an EAE benchmark, MAVEN-Arg offers three main advantages: (1) a comprehensive schema covering 162 event types and 612 argument roles, all with expert-written definitions and examples; (2) a large data scale, containing 98,591 events and 290,613 arguments obtained with laborious human annotation; and (3) the exhaustive annotation supporting all task variants of EAE.
  arXiv  Detail & Related papers  (2023-11-15T16:52:14Z)
• Abnormal Event Detection via Hypergraph Contrastive Learning [54.80429341415227]
  Abnormal event detection plays an important role in many real applications. In this paper, we study the unsupervised abnormal event detection problem in Attributed Heterogeneous Information Network. A novel hypergraph contrastive learning method, named AEHCL, is proposed to fully capture abnormal event patterns.
  arXiv  Detail & Related papers  (2023-04-02T08:23:20Z)
• Event Causality Extraction with Event Argument Correlations [13.403222002600558]
  Event Causality Extraction aims to extract cause-effect event causality pairs from plain texts. We propose a method with a dual grid tagging scheme to capture the intra- and inter-event argument correlations for ECE.
  arXiv  Detail & Related papers  (2023-01-27T09:48:31Z)
• Causal Knowledge Guided Societal Event Forecasting [24.437437565689393]
  We introduce a deep learning framework that integrates causal effect estimation into event forecasting. Two robust learning modules, including a feature reweighting module and an approximate loss, are introduced to enable prior knowledge injection.
  arXiv  Detail & Related papers  (2021-12-10T17:41:02Z)
• Learning Constraints and Descriptive Segmentation for Subevent Detection [74.48201657623218]
  We propose an approach to learning and enforcing constraints that capture dependencies between subevent detection and EventSeg prediction. We adopt Rectifier Networks for constraint learning and then convert the learned constraints to a regularization term in the loss function of the neural model.
  arXiv  Detail & Related papers  (2021-09-13T20:50:37Z)
• MAVEN: A Massive General Domain Event Detection Dataset [56.00401399384715]
  Event detection (ED) is the first and most fundamental step for extracting event knowledge from plain text. Existing datasets exhibit issues that limit further development of ED. We present a MAssive eVENt detection dataset (MAVEN), which contains 4,480 Wikipedia documents, 118,732 event mention instances, and 168 event types.
  arXiv  Detail & Related papers  (2020-04-28T15:25:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.