The Internal State of an LLM Knows When It's Lying

• URL: http://arxiv.org/abs/2304.13734v2
• Date: Tue, 17 Oct 2023 09:34:30 GMT
• Title: The Internal State of an LLM Knows When It's Lying
• Authors: Amos Azaria, Tom Mitchell
• Abstract summary: Large Language Models (LLMs) have shown exceptional performance in various tasks. One of their most prominent drawbacks is generating inaccurate or false information with a confident tone. We provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements.
• Score: 18.886091925252174
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: While Large Language Models (LLMs) have shown exceptional performance in various tasks, one of their most prominent drawbacks is generating inaccurate or false information with a confident tone. In this paper, we provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements. This includes both statements provided to the LLM, and statements that the LLM itself generates. Our approach is to train a classifier that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement. Experiments demonstrate that given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71\% to 83\% accuracy labeling which sentences are true versus false, depending on the LLM base model. Furthermore, we explore the relationship between our classifier's performance and approaches based on the probability assigned to the sentence by the LLM. We show that while LLM-assigned sentence probability is related to sentence truthfulness, this probability is also dependent on sentence length and the frequencies of words in the sentence, resulting in our trained classifier providing a more reliable approach to detecting truthfulness, highlighting its potential to enhance the reliability of LLM-generated content and its practical applicability in real-world scenarios.

Related papers

• Truth is Universal: Robust Detection of Lies in LLMs [18.13311575803723]
  Large Language Models (LLMs) have revolutionised natural language processing, exhibiting impressive human-like capabilities. In this work, we aim to develop a robust method to detect when an LLM is lying. We demonstrate the existence of a two-dimensional subspace, along which the activation vectors of true and false statements can be separated. This finding is universal and holds for various LLMs, including Gemma-7B, LLaMA2-13B, Mistral-7B and LLaMA3-8B. Our analysis explains the generalisation failures observed in previous studies and sets the stage for more
  arXiv  Detail & Related papers  (2024-07-03T13:01:54Z)
• A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagation [72.93327642336078]
  We propose Belief Tree Propagation (BTProp), a probabilistic framework for hallucination detection. BTProp introduces a belief tree of logically related statements by decomposing a parent statement into child statements. Our method improves baselines by 3%-9% (evaluated by AUROC and AUC-PR) on multiple hallucination detection benchmarks.
  arXiv  Detail & Related papers  (2024-06-11T05:21:37Z)
• CSS: Contrastive Semantic Similarity for Uncertainty Quantification of LLMs [1.515687944002438]
  We propose Contrastive Semantic Similarity, a module to obtain similarity features for measuring uncertainty for text pairs. We conduct extensive experiments with three large language models (LLMs) on several benchmark question-answering datasets. Results show that our proposed method performs better in estimating reliable responses of LLMs than comparable baselines.
  arXiv  Detail & Related papers  (2024-06-05T11:35:44Z)
• Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL [78.80673954827773]
  Large Language Models (LLMs) play a crucial role in capturing structured semantics to enhance language understanding, improve interpretability, and reduce bias. We propose using Semantic Role Labeling (SRL) as a fundamental task to explore LLMs' ability to extract structured semantics. We find interesting potential: LLMs can indeed capture semantic structures, and scaling-up doesn't always mirror potential. We are surprised to discover that significant overlap in the errors is made by both LLMs and untrained humans, accounting for almost 30% of all errors.
  arXiv  Detail & Related papers  (2024-05-10T11:44:05Z)
• $\forall$uto$\exists$val: Autonomous Assessment of LLMs in Formal Synthesis and Interpretation Tasks [21.12437562185667]
  This paper presents a new approach for scaling LLM assessment in translating formal syntax to natural language. We use context-free grammars (CFGs) to generate out-of-distribution datasets on the fly. We also conduct an assessment of several SOTA closed and open-source LLMs to showcase the feasibility and scalability of this paradigm.
  arXiv  Detail & Related papers  (2024-03-27T08:08:00Z)
• See the Unseen: Better Context-Consistent Knowledge-Editing by Noises [73.54237379082795]
  Knowledge-editing updates knowledge of large language models (LLMs) Existing works ignore this property and the editing lacks generalization. We empirically find that the effects of different contexts upon LLMs in recalling the same knowledge follow a Gaussian-like distribution.
  arXiv  Detail & Related papers  (2024-01-15T09:09:14Z)
• Assessing the Reliability of Large Language Model Knowledge [78.38870272050106]
  Large language models (LLMs) have been treated as knowledge bases due to their strong performance in knowledge probing tasks. How do we evaluate the capabilities of LLMs to consistently produce factually correct answers? We propose MOdel kNowledge relIabiliTy scORe (MONITOR), a novel metric designed to directly measure LLMs' factual reliability.
  arXiv  Detail & Related papers  (2023-10-15T12:40:30Z)
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models [79.01926242857613]
  Large language models (LLMs) are prone to hallucinations, generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts.
  arXiv  Detail & Related papers  (2023-09-07T17:45:31Z)
• Statistical Knowledge Assessment for Large Language Models [79.07989821512128]
  Given varying prompts regarding a factoid question, can a large language model (LLM) reliably generate factually correct answers? We propose KaRR, a statistical approach to assess factual knowledge for LLMs. Our results reveal that the knowledge in LLMs with the same backbone architecture adheres to the scaling law, while tuning on instruction-following data sometimes compromises the model's capability to generate factually correct text reliably.
  arXiv  Detail & Related papers  (2023-05-17T18:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.