The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets

• URL: http://arxiv.org/abs/2310.06824v3
• Date: Mon, 19 Aug 2024 01:18:41 GMT
• Title: The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
• Authors: Samuel Marks, Max Tegmark,
• Abstract summary: Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. We present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements.
• Score: 6.732432949368421
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have impressive capabilities, but are prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we use high-quality datasets of simple true/false statements to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements. We also show that simple difference-in-mean probes generalize as well as other probing techniques while identifying directions which are more causally implicated in model outputs.

Related papers

• Truth is Universal: Robust Detection of Lies in LLMs [18.13311575803723]
  Large Language Models (LLMs) have revolutionised natural language processing, exhibiting impressive human-like capabilities. In this work, we aim to develop a robust method to detect when an LLM is lying. We demonstrate the existence of a two-dimensional subspace, along which the activation vectors of true and false statements can be separated. This finding is universal and holds for various LLMs, including Gemma-7B, LLaMA2-13B, Mistral-7B and LLaMA3-8B. Our analysis explains the generalisation failures observed in previous studies and sets the stage for more
  arXiv  Detail & Related papers  (2024-07-03T13:01:54Z)
• FLAME: Factuality-Aware Alignment for Large Language Models [86.76336610282401]
  The conventional alignment process fails to enhance the factual accuracy of large language models (LLMs) We identify factors that lead to hallucination in both alignment steps: supervised fine-tuning (SFT) and reinforcement learning (RL) We propose factuality-aware alignment, comprised of factuality-aware SFT and factuality-aware RL through direct preference optimization.
  arXiv  Detail & Related papers  (2024-05-02T17:54:54Z)
• GRATH: Gradual Self-Truthifying for Large Language Models [63.502835648056305]
  GRAdual self-truTHifying (GRATH) is a novel post-processing method to enhance truthfulness of large language models (LLMs) GRATH iteratively refines truthfulness data and updates the model, leading to a gradual improvement in model truthfulness in a self-supervised manner. GRATH achieves state-of-the-art performance on TruthfulQA, with MC1 accuracy of 54.71% and MC2 accuracy of 69.10%, which even surpass those on 70B-LLMs.
  arXiv  Detail & Related papers  (2024-01-22T19:00:08Z)
• Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning [18.92421817900689]
  We introduce Truth Forest, a method that enhances truthfulness in large language models (LLMs) We also introduce Random Peek, a systematic technique considering an extended range of positions within the sequence.
  arXiv  Detail & Related papers  (2023-12-29T06:08:18Z)
• Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness? [53.98071556805525]
  Neural language models (LMs) can be used to evaluate the truth of factual statements. They can be queried for statement probabilities, or probed for internal representations of truthfulness. Past work has found that these two procedures sometimes disagree, and that probes tend to be more accurate than LM outputs. This has led some researchers to conclude that LMs "lie" or otherwise encode non-cooperative communicative intents.
  arXiv  Detail & Related papers  (2023-11-27T18:59:14Z)
• Personas as a Way to Model Truthfulness in Language Models [23.86655844340011]
  Large language models (LLMs) are trained on vast amounts of text from the internet. This paper presents an explanation for why LMs appear to know the truth despite not being trained with truth labels.
  arXiv  Detail & Related papers  (2023-10-27T14:27:43Z)
• FactCHD: Benchmarking Fact-Conflicting Hallucination Detection [64.4610684475899]
  FactCHD is a benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. We introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2.
  arXiv  Detail & Related papers  (2023-10-18T16:27:49Z)
• Do Large Language Models Know about Facts? [60.501902866946]
  Large language models (LLMs) have recently driven striking performance improvements across a range of natural language processing tasks. We aim to evaluate the extent and scope of factual knowledge within LLMs by designing the benchmark Pinocchio. Pinocchio contains 20K diverse factual questions that span different sources, timelines, domains, regions, and languages.
  arXiv  Detail & Related papers  (2023-10-08T14:26:55Z)
• How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions [34.53980255211931]
  Large language models (LLMs) can "lie", which we define as outputting false statements despite "knowing" the truth in a demonstrable sense. Here, we develop a simple lie detector that requires neither access to the LLM's activations nor ground-truth knowledge of the fact in question. Despite its simplicity, this lie detector is highly accurate and surprisingly general.
  arXiv  Detail & Related papers  (2023-09-26T16:07:54Z)
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models [79.01926242857613]
  Large language models (LLMs) are prone to hallucinations, generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts.
  arXiv  Detail & Related papers  (2023-09-07T17:45:31Z)
• The Internal State of an LLM Knows When It's Lying [18.886091925252174]
  Large Language Models (LLMs) have shown exceptional performance in various tasks. One of their most prominent drawbacks is generating inaccurate or false information with a confident tone. We provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements.
  arXiv  Detail & Related papers  (2023-04-26T02:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.