ClaimDiff: Comparing and Contrasting Claims on Contentious Issues
- URL: http://arxiv.org/abs/2205.12221v2
- Date: Sun, 11 Jun 2023 05:30:31 GMT
- Title: ClaimDiff: Comparing and Contrasting Claims on Contentious Issues
- Authors: Miyoung Ko, Ingyu Seong, Hwaran Lee, Joonsuk Park, Minsuk Chang,
Minjoon Seo
- Abstract summary: ClaimDiff is a dataset that primarily focuses on comparing the nuance between claim pairs.
We provide 2,941 annotated claim pairs from 268 news articles.
We observe that while humans are capable of detecting the nuances between claims, strong baselines struggle to detect them.
- Score: 23.260333361646495
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: With the growing importance of detecting misinformation, many studies have
focused on verifying factual claims by retrieving evidence. However, canonical
fact verification tasks do not apply to catching subtle differences in
factually consistent claims, which might still bias the readers, especially on
contentious political or economic issues. Our underlying assumption is that
among the trusted sources, one's argument is not necessarily more true than the
other, requiring comparison rather than verification. In this study, we propose
ClaimDiff, a novel dataset that primarily focuses on comparing the nuance
between claim pairs. In ClaimDiff, we provide 2,941 annotated claim pairs from
268 news articles. We observe that while humans are capable of detecting the
nuances between claims, strong baselines struggle to detect them, showing over
a 19% absolute gap with the humans. We hope this initial study could help
readers to gain an unbiased grasp of contentious issues through machine-aided
comparison.
Related papers
- Grounding Fallacies Misrepresenting Scientific Publications in Evidence [84.32990746227385]
We introduce MissciPlus, an extension of the fallacy detection dataset Missci.
MissciPlus builds on Missci by grounding the applied fallacies in real-world passages from misrepresented studies.
MissciPlus is the first logical fallacy dataset which pairs the real-world misrepresented evidence with incorrect claims.
arXiv Detail & Related papers (2024-08-23T03:16:26Z) - Missci: Reconstructing Fallacies in Misrepresented Science [84.32990746227385]
Health-related misinformation on social networks can lead to poor decision-making and real-world dangers.
Missci is a novel argumentation theoretical model for fallacious reasoning.
We present Missci as a dataset to test the critical reasoning abilities of large language models.
arXiv Detail & Related papers (2024-06-05T12:11:10Z) - What Makes Medical Claims (Un)Verifiable? Analyzing Entity and Relation
Properties for Fact Verification [8.086400003948143]
The BEAR-Fact corpus is the first corpus for scientific fact verification annotated with subject-relation-object triplets, evidence documents, and fact-checking verdicts.
We show that it is possible to reliably estimate the success of evidence retrieval purely from the claim text.
The dataset is available at http://www.ims.uni-stuttgart.de/data/bioclaim.
arXiv Detail & Related papers (2024-02-02T12:27:58Z) - Leveraging Social Discourse to Measure Check-worthiness of Claims for
Fact-checking [36.21314290592325]
We present CheckIt, a manually annotated large Twitter dataset for fine-grained claim check-worthiness.
We benchmark our dataset against a unified approach, CheckMate, that jointly determines whether a claim is check-worthy and the factors that led to that conclusion.
arXiv Detail & Related papers (2023-09-17T13:42:41Z) - Generating Literal and Implied Subquestions to Fact-check Complex Claims [64.81832149826035]
We focus on decomposing a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim.
We present ClaimDecomp, a dataset of decompositions for over 1000 claims.
We show that these subquestions can help identify relevant evidence to fact-check the full claim and derive the veracity through their answers.
arXiv Detail & Related papers (2022-05-14T00:40:57Z) - Robust Information Retrieval for False Claims with Distracting Entities
In Fact Extraction and Verification [2.624734563929267]
This paper shows that, compared with true claims, false claims more frequently contain irrelevant entities which can distract evidence retrieval model.
A BERT-based retrieval model made more mistakes in retrieving refuting evidence for false claims than supporting evidence for true claims.
arXiv Detail & Related papers (2021-12-10T17:11:50Z) - Topic-Aware Evidence Reasoning and Stance-Aware Aggregation for Fact
Verification [19.130541561303293]
We propose a novel topic-aware evidence reasoning and stance-aware aggregation model for fact verification.
Tests conducted on two benchmark datasets demonstrate the superiority of the proposed model over several state-of-the-art approaches for fact verification.
arXiv Detail & Related papers (2021-06-02T14:33:12Z) - AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs.
We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC.
We develop models for predicting veracity handling this ambiguity via soft labels.
arXiv Detail & Related papers (2021-04-01T17:40:08Z) - Misinformation Has High Perplexity [55.47422012881148]
We propose to leverage the perplexity to debunk false claims in an unsupervised manner.
First, we extract reliable evidence from scientific and news sources according to sentence similarity to the claims.
Second, we prime a language model with the extracted evidence and finally evaluate the correctness of given claims based on the perplexity scores at debunking time.
arXiv Detail & Related papers (2020-06-08T15:13:44Z) - Claim Check-Worthiness Detection as Positive Unlabelled Learning [53.24606510691877]
Claim check-worthiness detection is a critical component of fact checking systems.
We illuminate a central challenge in claim check-worthiness detection underlying all of these tasks.
Our best performing method is a unified approach which automatically corrects for this using a variant of positive unlabelled learning.
arXiv Detail & Related papers (2020-03-05T16:06:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.