Generating Scientific Claims for Zero-Shot Scientific Fact Checking

• URL: http://arxiv.org/abs/2203.12990v1
• Date: Thu, 24 Mar 2022 11:29:20 GMT
• Title: Generating Scientific Claims for Zero-Shot Scientific Fact Checking
• Authors: Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang
• Abstract summary: Automated scientific fact checking is difficult due to the complexity of scientific language and a lack of significant amounts of training data. We propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences. We also demonstrate its usefulness in zero-shot fact checking for biomedical claims.
• Score: 54.62086027306609
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated scientific fact checking is difficult due to the complexity of scientific language and a lack of significant amounts of training data, as annotation requires domain expertise. To address this challenge, we propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences, and demonstrate its usefulness in zero-shot fact checking for biomedical claims. We propose CLAIMGEN-BART, a new supervised method for generating claims supported by the literature, as well as KBIN, a novel method for generating claim negations. Additionally, we adapt an existing unsupervised entity-centric method of claim generation to biomedical claims, which we call CLAIMGEN-ENTITY. Experiments on zero-shot fact checking demonstrate that both CLAIMGEN-ENTITY and CLAIMGEN-BART, coupled with KBIN, achieve up to 90% performance of fully supervised models trained on manually annotated claims and evidence. A rigorous evaluation study demonstrates significant improvement in generated claim and negation quality over existing baselines

Related papers

• Robust Claim Verification Through Fact Detection [17.29665711917281]
  Our novel approach, FactDetect, leverages Large Language Models (LLMs) to generate concise factual statements from evidence. The generated facts are then combined with the claim and evidence. Our method demonstrates competitive results in the supervised claim verification model by 15% on the F1 score.
  arXiv  Detail & Related papers  (2024-07-25T20:03:43Z)
• Missci: Reconstructing Fallacies in Misrepresented Science [84.32990746227385]
  Health-related misinformation on social networks can lead to poor decision-making and real-world dangers. Missci is a novel argumentation theoretical model for fallacious reasoning. We present Missci as a dataset to test the critical reasoning abilities of large language models.
  arXiv  Detail & Related papers  (2024-06-05T12:11:10Z)
• SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables [68.76415918462418]
  We present SCITAB, a challenging evaluation dataset consisting of 1.2K expert-verified scientific claims. Through extensive evaluations, we demonstrate that SCITAB poses a significant challenge to state-of-the-art models. Our analysis uncovers several unique challenges posed by SCITAB, including table grounding, claim ambiguity, and compositional reasoning.
  arXiv  Detail & Related papers  (2023-05-22T16:13:50Z)
• SciFact-Open: Towards open-domain scientific claim verification [61.288725621156864]
  We present SciFact-Open, a new test collection designed to evaluate the performance of scientific claim verification systems. We collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models. We find that systems developed on smaller corpora struggle to generalize to SciFact-Open, exhibiting performance drops of at least 15 F1.
  arXiv  Detail & Related papers  (2022-10-25T05:45:00Z)
• SciClops: Detecting and Contextualizing Scientific Claims for Assisting Manual Fact-Checking [7.507186058512835]
  This paper describes SciClops, a method to help combat online scientific misinformation. SciClops involves three main steps to process scientific claims found in online news articles and social media postings. It effectively assists non-expert fact-checkers in the verification of complex scientific claims, outperforming commercial fact-checking systems.
  arXiv  Detail & Related papers  (2021-10-25T16:35:58Z)
• COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic [12.078052727772718]
  We introduce a FEVER-like dataset COVID-Fact of $4,086$ claims concerning the COVID-19 pandemic. The dataset contains claims, evidence for the claims, and contradictory claims refuted by the evidence.
  arXiv  Detail & Related papers  (2021-06-07T16:59:46Z)
• Misinformation Has High Perplexity [55.47422012881148]
  We propose to leverage the perplexity to debunk false claims in an unsupervised manner. First, we extract reliable evidence from scientific and news sources according to sentence similarity to the claims. Second, we prime a language model with the extracted evidence and finally evaluate the correctness of given claims based on the perplexity scores at debunking time.
  arXiv  Detail & Related papers  (2020-06-08T15:13:44Z)
• Fact or Fiction: Verifying Scientific Claims [53.29101835904273]
  We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that SUPPORTS or REFUTES a given scientific claim. We construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. We show that our system is able to verify claims related to COVID-19 by identifying evidence from the CORD-19 corpus.
  arXiv  Detail & Related papers  (2020-04-30T17:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.