What Makes Medical Claims (Un)Verifiable? Analyzing Entity and Relation Properties for Fact Verification

• URL: http://arxiv.org/abs/2402.01360v1
• Date: Fri, 2 Feb 2024 12:27:58 GMT
• Title: What Makes Medical Claims (Un)Verifiable? Analyzing Entity and Relation Properties for Fact Verification
• Authors: Amelie W\"uhrl and Yarik Menchaca Resendiz and Lara Grimminger and Roman Klinger
• Abstract summary: The BEAR-Fact corpus is the first corpus for scientific fact verification annotated with subject-relation-object triplets, evidence documents, and fact-checking verdicts. We show that it is possible to reliably estimate the success of evidence retrieval purely from the claim text. The dataset is available at http://www.ims.uni-stuttgart.de/data/bioclaim.
• Score: 8.086400003948143
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Biomedical claim verification fails if no evidence can be discovered. In these cases, the fact-checking verdict remains unknown and the claim is unverifiable. To improve upon this, we have to understand if there are any claim properties that impact its verifiability. In this work we assume that entities and relations define the core variables in a biomedical claim's anatomy and analyze if their properties help us to differentiate verifiable from unverifiable claims. In a study with trained annotation experts we prompt them to find evidence for biomedical claims, and observe how they refine search queries for their evidence search. This leads to the first corpus for scientific fact verification annotated with subject-relation-object triplets, evidence documents, and fact-checking verdicts (the BEAR-Fact corpus). We find (1) that discovering evidence for negated claims (e.g., X-does-not-cause-Y) is particularly challenging. Further, we see that annotators process queries mostly by adding constraints to the search and by normalizing entities to canonical names. (2) We compare our in-house annotations with a small crowdsourcing setting where we employ medical experts and laypeople. We find that domain expertise does not have a substantial effect on the reliability of annotations. Finally, (3), we demonstrate that it is possible to reliably estimate the success of evidence retrieval purely from the claim text~(.82\F), whereas identifying unverifiable claims proves more challenging (.27\F). The dataset is available at http://www.ims.uni-stuttgart.de/data/bioclaim.

Related papers

• Comparing Knowledge Sources for Open-Domain Scientific Claim Verification [6.726255259929497]
  We show that PubMed works better with specialized biomedical claims, while Wikipedia is more suited for everyday health concerns. We discuss the results, outline frequent retrieval patterns and challenges, and provide promising future directions.
  arXiv  Detail & Related papers  (2024-02-05T09:57:15Z)
• Complex Claim Verification with Evidence Retrieved in the Wild [73.19998942259073]
  We present the first fully automated pipeline to check real-world claims by retrieving raw evidence from the web. Our pipeline includes five components: claim decomposition, raw document retrieval, fine-grained evidence retrieval, claim-focused summarization, and veracity judgment.
  arXiv  Detail & Related papers  (2023-05-19T17:49:19Z)
• Decker: Double Check with Heterogeneous Knowledge for Commonsense Fact Verification [80.31112722910787]
  We propose Decker, a commonsense fact verification model that is capable of bridging heterogeneous knowledge. Experimental results on two commonsense fact verification benchmark datasets, CSQA2.0 and CREAK demonstrate the effectiveness of our Decker.
  arXiv  Detail & Related papers  (2023-05-10T06:28:16Z)
• Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidence [59.81749318292707]
  We propose a fact verification model named ReRead to retrieve evidence and verify claim. The proposed system is able to achieve significant improvements upon best-reported models under different settings.
  arXiv  Detail & Related papers  (2023-05-02T03:23:14Z)
• ClaimDiff: Comparing and Contrasting Claims on Contentious Issues [23.260333361646495]
  ClaimDiff is a dataset that primarily focuses on comparing the nuance between claim pairs. We provide 2,941 annotated claim pairs from 268 news articles. We observe that while humans are capable of detecting the nuances between claims, strong baselines struggle to detect them.
  arXiv  Detail & Related papers  (2022-05-24T17:29:13Z)
• Generating Literal and Implied Subquestions to Fact-check Complex Claims [64.81832149826035]
  We focus on decomposing a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim. We present ClaimDecomp, a dataset of decompositions for over 1000 claims. We show that these subquestions can help identify relevant evidence to fact-check the full claim and derive the veracity through their answers.
  arXiv  Detail & Related papers  (2022-05-14T00:40:57Z)
• Generating Scientific Claims for Zero-Shot Scientific Fact Checking [54.62086027306609]
  Automated scientific fact checking is difficult due to the complexity of scientific language and a lack of significant amounts of training data. We propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences. We also demonstrate its usefulness in zero-shot fact checking for biomedical claims.
  arXiv  Detail & Related papers  (2022-03-24T11:29:20Z)
• Topic-Aware Evidence Reasoning and Stance-Aware Aggregation for Fact Verification [19.130541561303293]
  We propose a novel topic-aware evidence reasoning and stance-aware aggregation model for fact verification. Tests conducted on two benchmark datasets demonstrate the superiority of the proposed model over several state-of-the-art approaches for fact verification.
  arXiv  Detail & Related papers  (2021-06-02T14:33:12Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)
• Explainable Automated Fact-Checking for Public Health Claims [11.529816799331979]
  We present the first study of explainable fact-checking for claims which require specific expertise. For our case study we choose the setting of public health. We explore two tasks: veracity prediction and explanation generation.
  arXiv  Detail & Related papers  (2020-10-19T23:51:33Z)
• DTCA: Decision Tree-based Co-Attention Networks for Explainable Claim Verification [16.144566353074314]
  We propose a Decision Tree-based Co-Attention model (DTCA) to discover evidence for explainable claim verification. Specifically, we first construct Decision Tree-based Evidence model (DTE) to select comments with high credibility as evidence in a transparent and interpretable way. We then design Co-attention Self-attention networks (CaSa) to make the selected evidence interact with claims.
  arXiv  Detail & Related papers  (2020-04-28T12:19:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.