The Case for Claim Difficulty Assessment in Automatic Fact Checking

• URL: http://arxiv.org/abs/2109.09689v1
• Date: Mon, 20 Sep 2021 16:59:50 GMT
• Title: The Case for Claim Difficulty Assessment in Automatic Fact Checking
• Authors: Prakhar Singh and Anubrata Das and Junyi Jessy Li and Matthew Lease
• Abstract summary: We argue that prediction of claim difficulty is a missing component of today's automated fact-checking architectures. We describe how this difficulty prediction task might be split into a set of distinct subtasks.
• Score: 18.230039157836888
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fact-checking is the process (human, automated, or hybrid) by which claims (i.e., purported facts) are evaluated for veracity. In this article, we raise an issue that has received little attention in prior work - that some claims are far more difficult to fact-check than others. We discuss the implications this has for both practical fact-checking and research on automated fact-checking, including task formulation and dataset design. We report a manual analysis undertaken to explore factors underlying varying claim difficulty and categorize several distinct types of difficulty. We argue that prediction of claim difficulty is a missing component of today's automated fact-checking architectures, and we describe how this difficulty prediction task might be split into a set of distinct subtasks.

Related papers

• Automated Justification Production for Claim Veracity in Fact Checking: A Survey on Architectures and Approaches [2.0140898354987353]
  Automated Fact-Checking (AFC) is the automated verification of claim accuracy. AFC is crucial in discerning truth from misinformation, especially given the huge amounts of content are generated online daily. Current research focuses on predicting claim veracity through metadata analysis and language scrutiny.
  arXiv  Detail & Related papers  (2024-07-09T01:54:13Z)
• How We Refute Claims: Automatic Fact-Checking through Flaw Identification and Explanation [4.376598435975689]
  This paper explores the novel task of flaw-oriented fact-checking, including aspect generation and flaw identification. We also introduce RefuteClaim, a new framework designed specifically for this task. Given the absence of an existing dataset, we present FlawCheck, a dataset created by extracting and transforming insights from expert reviews into relevant aspects and identified flaws.
  arXiv  Detail & Related papers  (2024-01-27T06:06:16Z)
• Generating Literal and Implied Subquestions to Fact-check Complex Claims [64.81832149826035]
  We focus on decomposing a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim. We present ClaimDecomp, a dataset of decompositions for over 1000 claims. We show that these subquestions can help identify relevant evidence to fact-check the full claim and derive the veracity through their answers.
  arXiv  Detail & Related papers  (2022-05-14T00:40:57Z)
• DialFact: A Benchmark for Fact-Checking in Dialogue [56.63709206232572]
  We construct DialFact, a benchmark dataset of 22,245 annotated conversational claims, paired with pieces of evidence from Wikipedia. We find that existing fact-checking models trained on non-dialogue data like FEVER fail to perform well on our task. We propose a simple yet data-efficient solution to effectively improve fact-checking performance in dialogue.
  arXiv  Detail & Related papers  (2021-10-15T17:34:35Z)
• A Survey on Automated Fact-Checking [18.255327608480165]
  We survey automated fact-checking stemming from natural language processing, and discuss its connections to related tasks and disciplines. We present an overview of existing datasets and models, aiming to unify the various definitions given and identify common concepts.
  arXiv  Detail & Related papers  (2021-08-26T16:34:51Z)
• FaVIQ: FAct Verification from Information-seeking Questions [77.7067957445298]
  We construct a large-scale fact verification dataset called FaVIQ using information-seeking questions posed by real users. Our claims are verified to be natural, contain little lexical bias, and require a complete understanding of the evidence for verification.
  arXiv  Detail & Related papers  (2021-07-05T17:31:44Z)
• Competency Problems: On Finding and Removing Artifacts in Language Data [50.09608320112584]
  We argue that for complex language understanding tasks, all simple feature correlations are spurious. We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account.
  arXiv  Detail & Related papers  (2021-04-17T21:34:10Z)
• A Review on Fact Extraction and Verification [19.373340472113703]
  We study the fact checking problem, which aims to identify the veracity of a given claim. We focus on the task of Fact Extraction and VERification (FEVER) and its accompanied dataset. This task is essential and can be the building block of applications such as fake news detection and medical claim verification.
  arXiv  Detail & Related papers  (2020-10-06T20:05:43Z)
• Generating Fact Checking Explanations [52.879658637466605]
  A crucial piece of the puzzle that is still missing is to understand how to automate the most elaborate part of the process. This paper provides the first study of how these explanations can be generated automatically based on available claim context. Our results indicate that optimising both objectives at the same time, rather than training them separately, improves the performance of a fact checking system.
  arXiv  Detail & Related papers  (2020-04-13T05:23:25Z)
• Claim Check-Worthiness Detection as Positive Unlabelled Learning [53.24606510691877]
  Claim check-worthiness detection is a critical component of fact checking systems. We illuminate a central challenge in claim check-worthiness detection underlying all of these tasks. Our best performing method is a unified approach which automatically corrects for this using a variant of positive unlabelled learning.
  arXiv  Detail & Related papers  (2020-03-05T16:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.