FactGenius: Combining Zero-Shot Prompting and Fuzzy Relation Mining to Improve Fact Verification with Knowledge Graphs

• URL: http://arxiv.org/abs/2406.01311v1
• Date: Mon, 3 Jun 2024 13:24:37 GMT
• Title: FactGenius: Combining Zero-Shot Prompting and Fuzzy Relation Mining to Improve Fact Verification with Knowledge Graphs
• Authors: Sushant Gautam,
• Abstract summary: We present FactGenius, a novel method that enhances fact-checking by combining zero-shot prompting of large language models with fuzzy text matching on knowledge graphs. The evaluation of FactGenius on the FactKG, a benchmark dataset for fact verification, demonstrates that it significantly outperforms existing baselines.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fact-checking is a crucial natural language processing (NLP) task that verifies the truthfulness of claims by considering reliable evidence. Traditional methods are often limited by labour-intensive data curation and rule-based approaches. In this paper, we present FactGenius, a novel method that enhances fact-checking by combining zero-shot prompting of large language models (LLMs) with fuzzy text matching on knowledge graphs (KGs). Leveraging DBpedia, a structured linked data dataset derived from Wikipedia, FactGenius refines LLM-generated connections using similarity measures to ensure accuracy. The evaluation of FactGenius on the FactKG, a benchmark dataset for fact verification, demonstrates that it significantly outperforms existing baselines, particularly when fine-tuning RoBERTa as a classifier. The two-stage approach of filtering and validating connections proves crucial, achieving superior performance across various reasoning types and establishing FactGenius as a promising tool for robust fact-checking. The code and materials are available at https://github.com/SushantGautam/FactGenius.

Related papers

• Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model [6.097530398802087]
  This paper explores the correlation between the degree of noise and its impact on language models through instruction tuning. Specifically, we found multiple intriguing findings of the correlation between the factuality of the dataset and instruction tuning.
  arXiv  Detail & Related papers  (2024-04-15T12:20:09Z)
• Heterogeneous Graph Reasoning for Fact Checking over Texts and Tables [22.18384189336634]
  HeterFC is a word-level Heterogeneous-graph-based model for Fact Checking over unstructured and structured information. We perform information propagation via a relational graph neural network, interactions between claims and evidence. We introduce a multitask loss function to account for potential inaccuracies in evidence retrieval.
  arXiv  Detail & Related papers  (2024-02-20T14:10:40Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)
• EXPLAIN, EDIT, GENERATE: Rationale-Sensitive Counterfactual Data Augmentation for Multi-hop Fact Verification [28.453817513380276]
  We develop a rationale-sensitive method to generate linguistically diverse and label-flipping counterfactuals. In specific, the diverse and fluent counterfactuals are generated via an Explain-Edit-Generate architecture. Experimental results show that the proposed approach outperforms the SOTA baselines.
  arXiv  Detail & Related papers  (2023-10-23T02:39:14Z)
• Generating Benchmarks for Factuality Evaluation of Language Models [61.69950787311278]
  We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score and perplexity do not always agree on model ranking; (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation.
  arXiv  Detail & Related papers  (2023-07-13T17:14:38Z)
• $\texttt{causalAssembly}$: Generating Realistic Production Data for Benchmarking Causal Discovery [1.3048920509133808]
  We build a system for generation of semisynthetic manufacturing data that supports benchmarking of causal discovery methods. We employ distributional random forests to flexibly estimate and represent conditional distributions. Using the library, we showcase how to benchmark several well-known causal discovery algorithms.
  arXiv  Detail & Related papers  (2023-06-19T10:05:54Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [50.62245481416744]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)
• WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning [40.5830891229718]
  We propose a weakly supervised framework that aggregates multiple resources to train a precise and efficient factual metric, namely WeCheck. Comprehensive experiments on a variety of tasks demonstrate the strong performance of WeCheck, which achieves a 3.4% absolute improvement over previous state-of-the-art methods on TRUE benchmark on average.
  arXiv  Detail & Related papers  (2022-12-20T08:04:36Z)
• Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization [63.21819285337555]
  We show that NLI models can be effective for this task when the training data is augmented with high-quality task-oriented examples. We introduce Falsesum, a data generation pipeline leveraging a controllable text generation model to perturb human-annotated summaries. We show that models trained on a Falsesum-augmented NLI dataset improve the state-of-the-art performance across four benchmarks for detecting factual inconsistency in summarization.
  arXiv  Detail & Related papers  (2022-05-12T10:43:42Z)
• FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations [114.94628499698096]
  We propose FactGraph, a method that decomposes the document and the summary into structured meaning representations (MRs) MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. Experiments on different benchmarks for evaluating factuality show that FactGraph outperforms previous approaches by up to 15%.
  arXiv  Detail & Related papers  (2022-04-13T16:45:33Z)
• A Multi-Level Attention Model for Evidence-Based Fact Checking [58.95413968110558]
  We present a simple model that can be trained on sequence structures. Results on a large-scale dataset for Fact Extraction and VERification show that our model outperforms the graph-based approaches.
  arXiv  Detail & Related papers  (2021-06-02T05:40:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.