Evidence-based Interpretable Open-domain Fact-checking with Large Language Models

• URL: http://arxiv.org/abs/2312.05834v1
• Date: Sun, 10 Dec 2023 09:27:50 GMT
• Title: Evidence-based Interpretable Open-domain Fact-checking with Large Language Models
• Authors: Xin Tan, Bowei Zou and Ai Ti Aw
• Abstract summary: We introduce the Open-domain Explainable Fact-checking (OE-Fact) system for claim-checking in real-world scenarios. The OE-Fact system can leverage the powerful understanding and reasoning capabilities of large language models (LLMs) to validate claims. Experimental results show that our OE-Fact system outperforms general fact-checking baseline systems in both closed- and open-domain scenarios.
• Score: 26.89527395822654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Universal fact-checking systems for real-world claims face significant challenges in gathering valid and sufficient real-time evidence and making reasoned decisions. In this work, we introduce the Open-domain Explainable Fact-checking (OE-Fact) system for claim-checking in real-world scenarios. The OE-Fact system can leverage the powerful understanding and reasoning capabilities of large language models (LLMs) to validate claims and generate causal explanations for fact-checking decisions. To adapt the traditional three-module fact-checking framework to the open domain setting, we first retrieve claim-related information as relevant evidence from open websites. After that, we retain the evidence relevant to the claim through LLM and similarity calculation for subsequent verification. We evaluate the performance of our adapted three-module OE-Fact system on the Fact Extraction and Verification (FEVER) dataset. Experimental results show that our OE-Fact system outperforms general fact-checking baseline systems in both closed- and open-domain scenarios, ensuring stable and accurate verdicts while providing concise and convincing real-time explanations for fact-checking decisions.

Related papers

• Augmenting the Veracity and Explanations of Complex Fact Checking via Iterative Self-Revision with LLMs [10.449165630417522]
  We construct two complex fact-checking datasets in the Chinese scenarios: CHEF-EG and TrendFact. These datasets involve complex facts in areas such as health, politics, and society. We propose a unified framework called FactISR to perform mutual feedback between veracity and explanations.
  arXiv  Detail & Related papers  (2024-10-19T15:25:19Z)
• OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs [27.89053798151106]
  OpenFactCheck is a unified factuality evaluation framework for large language models. OpenFactCheck consists of three modules: (i) CUSTCHECKER, (ii) LLMEVAL, and (iii) CHECKEREVAL.
  arXiv  Detail & Related papers  (2024-05-09T07:15:19Z)
• RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict [34.2739191920746]
  High-quality evidence plays a vital role in enhancing fact-checking systems. We propose a method based on a Large Language Model to automatically retrieve and summarize evidence from the Web. We construct RU22Fact, a novel explainable fact-checking dataset on the Russia-Ukraine conflict in 2022 of 16K samples.
  arXiv  Detail & Related papers  (2024-03-25T11:56:29Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)
• EX-FEVER: A Dataset for Multi-hop Explainable Fact Verification [22.785622371421876]
  We present a pioneering dataset for multi-hop explainable fact verification. With over 60,000 claims involving 2-hop and 3-hop reasoning, each is created by summarizing and modifying information from hyperlinked Wikipedia documents. We demonstrate a novel baseline system on our EX-FEVER dataset, showcasing document retrieval, explanation generation, and claim verification.
  arXiv  Detail & Related papers  (2023-10-15T06:46:15Z)
• Give Me More Details: Improving Fact-Checking with Latent Retrieval [58.706972228039604]
  Evidence plays a crucial role in automated fact-checking. Existing fact-checking systems either assume the evidence sentences are given or use the search snippets returned by the search engine. We propose to incorporate full text from source documents as evidence and introduce two enriched datasets.
  arXiv  Detail & Related papers  (2023-05-25T15:01:19Z)
• WiCE: Real-World Entailment for Claims in Wikipedia [63.234352061821625]
  We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim. We show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.
  arXiv  Detail & Related papers  (2023-03-02T17:45:32Z)
• CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking [55.75590135151682]
  CHEF is the first CHinese Evidence-based Fact-checking dataset of 10K real-world claims. The dataset covers multiple domains, ranging from politics to public health, and provides annotated evidence retrieved from the Internet.
  arXiv  Detail & Related papers  (2022-06-06T09:11:03Z)
• DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking [46.13738685855884]
  We show that current systems for fact-checking are vulnerable to three categories of realistic challenges for fact-checking. We present a system designed to be resilient to these "attacks" using multiple pointer networks for document selection. We find that in handling these attacks we obtain state-of-the-art results on FEVER, largely due to improved evidence retrieval.
  arXiv  Detail & Related papers  (2020-04-27T15:18:49Z)
• Generating Fact Checking Explanations [52.879658637466605]
  A crucial piece of the puzzle that is still missing is to understand how to automate the most elaborate part of the process. This paper provides the first study of how these explanations can be generated automatically based on available claim context. Our results indicate that optimising both objectives at the same time, rather than training them separately, improves the performance of a fact checking system.
  arXiv  Detail & Related papers  (2020-04-13T05:23:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.