FactLLaMA: Optimizing Instruction-Following Language Models with
External Knowledge for Automated Fact-Checking
- URL: http://arxiv.org/abs/2309.00240v1
- Date: Fri, 1 Sep 2023 04:14:39 GMT
- Title: FactLLaMA: Optimizing Instruction-Following Language Models with
External Knowledge for Automated Fact-Checking
- Authors: Tsun-Hin Cheung and Kin-Man Lam
- Abstract summary: We propose combining the power of instruction-following language models with external evidence retrieval to enhance fact-checking performance.
Our approach involves leveraging search engines to retrieve relevant evidence for a given input claim.
Then, we instruct-tune an open-sourced language model, called LLaMA, using this evidence, enabling it to predict the veracity of the input claim more accurately.
- Score: 10.046323978189847
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Automatic fact-checking plays a crucial role in combating the spread of
misinformation. Large Language Models (LLMs) and Instruction-Following
variants, such as InstructGPT and Alpaca, have shown remarkable performance in
various natural language processing tasks. However, their knowledge may not
always be up-to-date or sufficient, potentially leading to inaccuracies in
fact-checking. To address this limitation, we propose combining the power of
instruction-following language models with external evidence retrieval to
enhance fact-checking performance. Our approach involves leveraging search
engines to retrieve relevant evidence for a given input claim. This external
evidence serves as valuable supplementary information to augment the knowledge
of the pretrained language model. Then, we instruct-tune an open-sourced
language model, called LLaMA, using this evidence, enabling it to predict the
veracity of the input claim more accurately. To evaluate our method, we
conducted experiments on two widely used fact-checking datasets: RAWFC and
LIAR. The results demonstrate that our approach achieves state-of-the-art
performance in fact-checking tasks. By integrating external evidence, we bridge
the gap between the model's knowledge and the most up-to-date and sufficient
context available, leading to improved fact-checking outcomes. Our findings
have implications for combating misinformation and promoting the dissemination
of accurate information on online platforms. Our released materials are
accessible at: https://thcheung.github.io/factllama.
Related papers
- ZeFaV: Boosting Large Language Models for Zero-shot Fact Verification [2.6874004806796523]
ZeFaV is a zero-shot based fact-checking verification framework to enhance the performance on fact verification task of large language models.
We conducted empirical experiments to evaluate our approach on two multi-hop fact-checking datasets including HoVer and FEVEROUS.
arXiv Detail & Related papers (2024-11-18T02:35:15Z) - Fact or Fiction? Improving Fact Verification with Knowledge Graphs through Simplified Subgraph Retrievals [0.0]
We present efficient methods for verifying claims on a dataset where the evidence is in the form of structured knowledge graphs.
By simplifying the evidence retrieval process, we are able to construct models that both require less computational resources and achieve better test-set accuracy.
arXiv Detail & Related papers (2024-08-14T10:46:15Z) - Large Language Models are Limited in Out-of-Context Knowledge Reasoning [65.72847298578071]
Large Language Models (LLMs) possess extensive knowledge and strong capabilities in performing in-context reasoning.
This paper focuses on a significant aspect of out-of-context reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine multiple knowledge to infer new knowledge.
arXiv Detail & Related papers (2024-06-11T15:58:59Z) - Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models [75.75038268227554]
Self-Checker is a framework comprising a set of plug-and-play modules that facilitate fact-checking.
This framework provides a fast and efficient way to construct fact-checking systems in low-resource environments.
arXiv Detail & Related papers (2023-05-24T01:46:07Z) - Context-faithful Prompting for Large Language Models [51.194410884263135]
Large language models (LLMs) encode parametric knowledge about world facts.
Their reliance on parametric knowledge may cause them to overlook contextual cues, leading to incorrect predictions in context-sensitive NLP tasks.
We assess and enhance LLMs' contextual faithfulness in two aspects: knowledge conflict and prediction with abstention.
arXiv Detail & Related papers (2023-03-20T17:54:58Z) - Large Language Models with Controllable Working Memory [64.71038763708161]
Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP)
What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining.
How the model's world knowledge interacts with the factual information presented in the context remains under explored.
arXiv Detail & Related papers (2022-11-09T18:58:29Z) - LM-CORE: Language Models with Contextually Relevant External Knowledge [13.451001884972033]
We argue that storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of knowledge and resource requirements.
We present LM-CORE -- a general framework to achieve this -- that allows textitdecoupling of the language model training from the external knowledge source.
Experimental results show that LM-CORE, having access to external knowledge, achieves significant and robust outperformance over state-of-the-art knowledge-enhanced language models on knowledge probing tasks.
arXiv Detail & Related papers (2022-08-12T18:59:37Z) - CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking [55.75590135151682]
CHEF is the first CHinese Evidence-based Fact-checking dataset of 10K real-world claims.
The dataset covers multiple domains, ranging from politics to public health, and provides annotated evidence retrieved from the Internet.
arXiv Detail & Related papers (2022-06-06T09:11:03Z) - Language Models as Fact Checkers? [39.29607585655352]
We create an effective end-to-end fact checker using a solely a language model.
We show that our zero-shot LM approach outperforms a random baseline on the standard FEVER task.
arXiv Detail & Related papers (2020-06-07T09:52:05Z) - Generating Fact Checking Explanations [52.879658637466605]
A crucial piece of the puzzle that is still missing is to understand how to automate the most elaborate part of the process.
This paper provides the first study of how these explanations can be generated automatically based on available claim context.
Our results indicate that optimising both objectives at the same time, rather than training them separately, improves the performance of a fact checking system.
arXiv Detail & Related papers (2020-04-13T05:23:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.