Robust Claim Verification Through Fact Detection

• URL: http://arxiv.org/abs/2407.18367v1
• Date: Thu, 25 Jul 2024 20:03:43 GMT
• Title: Robust Claim Verification Through Fact Detection
• Authors: Nazanin Jafari, James Allan,
• Abstract summary: Our novel approach, FactDetect, leverages Large Language Models (LLMs) to generate concise factual statements from evidence. The generated facts are then combined with the claim and evidence. Our method demonstrates competitive results in the supervised claim verification model by 15% on the F1 score.
• Score: 17.29665711917281
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Claim verification can be a challenging task. In this paper, we present a method to enhance the robustness and reasoning capabilities of automated claim verification through the extraction of short facts from evidence. Our novel approach, FactDetect, leverages Large Language Models (LLMs) to generate concise factual statements from evidence and label these facts based on their semantic relevance to the claim and evidence. The generated facts are then combined with the claim and evidence. To train a lightweight supervised model, we incorporate a fact-detection task into the claim verification process as a multitasking approach to improve both performance and explainability. We also show that augmenting FactDetect in the claim verification prompt enhances performance in zero-shot claim verification using LLMs. Our method demonstrates competitive results in the supervised claim verification model by 15% on the F1 score when evaluated for challenging scientific claim verification datasets. We also demonstrate that FactDetect can be augmented with claim and evidence for zero-shot prompting (AugFactDetect) in LLMs for verdict prediction. We show that AugFactDetect outperforms the baseline with statistical significance on three challenging scientific claim verification datasets with an average of 17.3% performance gain compared to the best performing baselines.

Related papers

• ZeFaV: Boosting Large Language Models for Zero-shot Fact Verification [2.6874004806796523]
  ZeFaV is a zero-shot based fact-checking verification framework to enhance the performance on fact verification task of large language models. We conducted empirical experiments to evaluate our approach on two multi-hop fact-checking datasets including HoVer and FEVEROUS.
  arXiv  Detail & Related papers  (2024-11-18T02:35:15Z)
• FactLens: Benchmarking Fine-Grained Fact Verification [6.814173254027381]
  We advocate for a shift toward fine-grained verification, where complex claims are broken down into smaller sub-claims for individual verification. We introduce FactLens, a benchmark for evaluating fine-grained fact verification, with metrics and automated evaluators of sub-claim quality. Our results show alignment between automated FactLens evaluators and human judgments, and we discuss the impact of sub-claim characteristics on the overall verification performance.
  arXiv  Detail & Related papers  (2024-11-08T21:26:57Z)
• Multimodal Misinformation Detection using Large Vision-Language Models [7.505532091249881]
  Large language models (LLMs) have shown remarkable performance in various tasks. Few approaches consider evidence retrieval as part of misinformation detection. We propose a novel re-ranking approach for multimodal evidence retrieval.
  arXiv  Detail & Related papers  (2024-07-19T13:57:11Z)
• RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict [34.2739191920746]
  High-quality evidence plays a vital role in enhancing fact-checking systems. We propose a method based on a Large Language Model to automatically retrieve and summarize evidence from the Web. We construct RU22Fact, a novel explainable fact-checking dataset on the Russia-Ukraine conflict in 2022 of 16K samples.
  arXiv  Detail & Related papers  (2024-03-25T11:56:29Z)
• Give Me More Details: Improving Fact-Checking with Latent Retrieval [58.706972228039604]
  Evidence plays a crucial role in automated fact-checking. Existing fact-checking systems either assume the evidence sentences are given or use the search snippets returned by the search engine. We propose to incorporate full text from source documents as evidence and introduce two enriched datasets.
  arXiv  Detail & Related papers  (2023-05-25T15:01:19Z)
• Read it Twice: Towards Faithfully Interpretable Fact Verification by Revisiting Evidence [59.81749318292707]
  We propose a fact verification model named ReRead to retrieve evidence and verify claim. The proposed system is able to achieve significant improvements upon best-reported models under different settings.
  arXiv  Detail & Related papers  (2023-05-02T03:23:14Z)
• Generating Scientific Claims for Zero-Shot Scientific Fact Checking [54.62086027306609]
  Automated scientific fact checking is difficult due to the complexity of scientific language and a lack of significant amounts of training data. We propose scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences. We also demonstrate its usefulness in zero-shot fact checking for biomedical claims.
  arXiv  Detail & Related papers  (2022-03-24T11:29:20Z)
• A Multi-Level Attention Model for Evidence-Based Fact Checking [58.95413968110558]
  We present a simple model that can be trained on sequence structures. Results on a large-scale dataset for Fact Extraction and VERification show that our model outperforms the graph-based approaches.
  arXiv  Detail & Related papers  (2021-06-02T05:40:12Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)
• Generating Fact Checking Explanations [52.879658637466605]
  A crucial piece of the puzzle that is still missing is to understand how to automate the most elaborate part of the process. This paper provides the first study of how these explanations can be generated automatically based on available claim context. Our results indicate that optimising both objectives at the same time, rather than training them separately, improves the performance of a fact checking system.
  arXiv  Detail & Related papers  (2020-04-13T05:23:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.