CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media

• URL: http://arxiv.org/abs/2210.04447v1
• Date: Mon, 10 Oct 2022 06:05:52 GMT
• Title: CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media
• Authors: Momchil Hardalov, Anton Chernyavskiy, Ivan Koychev, Dmitry Ilvovsky, Preslav Nakov
• Abstract summary: We propose an end-to-end framework to learn from noisy data based on modified self-adaptive training. Our experiments on the CLEF'21 CheckThat! test set show improvements over the state of the art by two points absolute.
• Score: 19.688259030184508
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While there has been substantial progress in developing systems to automate fact-checking, they still lack credibility in the eyes of the users. Thus, an interesting approach has emerged: to perform automatic fact-checking by verifying whether an input claim has been previously fact-checked by professional fact-checkers and to return back an article that explains their decision. This is a sensible approach as people trust manual fact-checking, and as many claims are repeated multiple times. Yet, a major issue when building such systems is the small number of known tweet--verifying article pairs available for training. Here, we aim to bridge this gap by making use of crowd fact-checking, i.e., mining claims in social media for which users have responded with a link to a fact-checking article. In particular, we mine a large-scale collection of 330,000 tweets paired with a corresponding fact-checking article. We further propose an end-to-end framework to learn from this noisy data based on modified self-adaptive training, in a distant supervision scenario. Our experiments on the CLEF'21 CheckThat! test set show improvements over the state of the art by two points absolute. Our code and datasets are available at https://github.com/mhardalov/crowdchecked-claims

Related papers

• Crowd Intelligence for Early Misinformation Prediction on Social Media [29.494819549803772]
  We introduce CROWDSHIELD, a crowd intelligence-based method for early misinformation prediction. We employ Q-learning to capture the two dimensions -- stances and claims. We propose MIST, a manually annotated misinformation detection Twitter corpus.
  arXiv  Detail & Related papers  (2024-08-08T13:45:23Z)
• OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs [64.25176233153657]
  OpenFactCheck is an open-sourced fact-checking framework for large language models. It allows users to easily customize an automatic fact-checking system. It also assesses the factuality of all claims in an input document using that system.
  arXiv  Detail & Related papers  (2024-08-06T15:49:58Z)
• Fact Checking Beyond Training Set [64.88575826304024]
  We show that the retriever-reader suffers from performance deterioration when it is trained on labeled data from one domain and used in another domain. We propose an adversarial algorithm to make the retriever component robust against distribution shift. We then construct eight fact checking scenarios from these datasets, and compare our model to a set of strong baseline models.
  arXiv  Detail & Related papers  (2024-03-27T15:15:14Z)
• MythQA: Query-Based Large-Scale Check-Worthy Claim Detection through Multi-Answer Open-Domain Question Answering [8.70509665552136]
  Check-worthy claim detection aims at providing plausible misinformation to downstream fact-checking systems or human experts to check. Many efforts have been put into how to identify check-worthy claims from a small scale of pre-collected claims, but how to efficiently detect check-worthy claims directly from a large-scale information source, such as Twitter, remains underexplored. We introduce MythQA, a new multi-answer open-domain question answering(QA) task that involves contradictory stance mining for query-based large-scale check-worthy claim detection.
  arXiv  Detail & Related papers  (2023-07-21T18:35:24Z)
• Harnessing Abstractive Summarization for Fact-Checked Claim Detection [8.49182897482236]
  Social media platforms have become new battlegrounds for anti-social elements, with misinformation being the weapon of choice. We believe that the solution lies in partial automation of the fact-checking life cycle, saving human time for tasks which require high cognition. We propose a new workflow for efficiently detecting previously fact-checked claims that uses abstractive summarization to generate crisp queries.
  arXiv  Detail & Related papers  (2022-09-10T07:32:36Z)
• Synthetic Disinformation Attacks on Automated Fact Verification Systems [53.011635547834025]
  We explore the sensitivity of automated fact-checkers to synthetic adversarial evidence in two simulated settings. We show that these systems suffer significant performance drops against these attacks. We discuss the growing threat of modern NLG systems as generators of disinformation.
  arXiv  Detail & Related papers  (2022-02-18T19:01:01Z)
• FaVIQ: FAct Verification from Information-seeking Questions [77.7067957445298]
  We construct a large-scale fact verification dataset called FaVIQ using information-seeking questions posed by real users. Our claims are verified to be natural, contain little lexical bias, and require a complete understanding of the evidence for verification.
  arXiv  Detail & Related papers  (2021-07-05T17:31:44Z)
• Generating Fact Checking Briefs [97.82546239639964]
  We investigate how to increase the accuracy and efficiency of fact checking by providing information about the claim before performing the check. We develop QABriefer, a model that generates a set of questions conditioned on the claim, searches the web for evidence, and generates answers. We show that fact checking with briefs -- in particular QABriefs -- increases the accuracy of crowdworkers by 10% while slightly decreasing the time taken.
  arXiv  Detail & Related papers  (2020-11-10T23:02:47Z)
• Team Alex at CLEF CheckThat! 2020: Identifying Check-Worthy Tweets With Transformer Models [28.25006244616817]
  We propose a model for detecting check-worthy tweets about COVID-19, which combines deep contextualized text representations with modeling the social context of the tweet. Our official submission to the English version of CLEF-2020 CheckThat! Task 1, system Team_Alex, was ranked second with a MAP score of 0.8034.
  arXiv  Detail & Related papers  (2020-09-07T08:03:21Z)
• That is a Known Lie: Detecting Previously Fact-Checked Claims [34.30218503006579]
  A large number of fact-checked claims have been accumulated. Politicians like to repeat their favorite statements, true or false, over and over again. It is important to try to save this effort and to avoid wasting time on claims that have already been fact-checked.
  arXiv  Detail & Related papers  (2020-05-12T21:25:37Z)
• Claim Check-Worthiness Detection as Positive Unlabelled Learning [53.24606510691877]
  Claim check-worthiness detection is a critical component of fact checking systems. We illuminate a central challenge in claim check-worthiness detection underlying all of these tasks. Our best performing method is a unified approach which automatically corrects for this using a variant of positive unlabelled learning.
  arXiv  Detail & Related papers  (2020-03-05T16:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.