Measuring Faithfulness of Chains of Thought by Unlearning Reasoning Steps

• URL: http://arxiv.org/abs/2502.14829v1
• Date: Thu, 20 Feb 2025 18:45:05 GMT
• Title: Measuring Faithfulness of Chains of Thought by Unlearning Reasoning Steps
• Authors: Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasović, Yonatan Belinkov,
• Abstract summary: We present Faithfulness by Unlearning Reasoning steps (FUR), a framework for measuring parametric faithfulness of generated reasoning. FUR erases information contained in reasoning steps from model parameters. We show that FUR is frequently able to change the underlying models' prediction by unlearning key steps, indicating when a CoT is parametrically faithful.
• Score: 32.60500043302544
• License:
• Abstract: When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. However, despite much work on CoT prompting, it is unclear if CoT reasoning is faithful to the models' parameteric beliefs. We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework. FUR erases information contained in reasoning steps from model parameters. We perform experiments unlearning CoTs of four LMs prompted on four multi-choice question answering (MCQA) datasets. Our experiments show that FUR is frequently able to change the underlying models' prediction by unlearning key steps, indicating when a CoT is parametrically faithful. Further analysis shows that CoTs generated by models post-unlearning support different answers, hinting at a deeper effect of unlearning. Importantly, CoT steps identified as important by FUR do not align well with human notions of plausbility, emphasizing the need for specialized alignment

Related papers

• Understanding Chain-of-Thought in LLMs through Information Theory [16.78730663293352]
  We formalize Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs) through an information-theoretic lens. Specifically, our framework quantifies the information gain' at each reasoning step, enabling the identification of failure modes. We demonstrate the efficacy of our approach through extensive experiments on toy and GSM-8K data, where it significantly outperforms existing outcome-based methods.
  arXiv  Detail & Related papers  (2024-11-18T19:14:36Z)
• Improve Vision Language Model Chain-of-thought Reasoning [86.83335752119741]
  Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. We show that training VLM on short answers does not generalize well to reasoning tasks that require more detailed responses.
  arXiv  Detail & Related papers  (2024-10-21T17:00:06Z)
• Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods [59.779795063072655]
  Chain-of-Thought (CoT) prompting and its variants have gained popularity as effective methods for solving multi-step reasoning problems. We analyze CoT prompting from a statistical estimation perspective, providing a comprehensive characterization of its sample complexity.
  arXiv  Detail & Related papers  (2024-08-25T04:07:18Z)
• Chain-of-Probe: Examing the Necessity and Accuracy of CoT Step-by-Step [81.50681925980135]
  We propose a method to probe changes in the mind during the model's reasoning. By analyzing patterns in mind change, we examine the correctness of the model's reasoning. Our validation reveals that many responses, although correct in their final answer, contain errors in their reasoning process.
  arXiv  Detail & Related papers  (2024-06-23T15:50:22Z)
• A Hopfieldian View-based Interpretation for Chain-of-Thought Reasoning [48.51969964676017]
  Chain-of-Thought (CoT) holds a significant place in augmenting the reasoning performance for large language models. We propose a Read-and-Control approach for controlling the accuracy of CoT.
  arXiv  Detail & Related papers  (2024-06-18T04:07:13Z)
• A Fixed-Point Approach for Causal Generative Modeling [20.88890689294816]
  We propose a novel formalism for describing Structural Causal Models (SCMs) as fixed-point problems on causally ordered variables. We establish the weakest known conditions for their unique recovery given the topological ordering (TO)
  arXiv  Detail & Related papers  (2024-04-10T12:29:05Z)
• Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs [52.42505579545893]
  Large language models (LLMs) demonstrate strong reasoning abilities when prompted to generate chain-of-thought explanations alongside answers. We propose a novel discriminative and generative CoT evaluation paradigm to assess LLMs' knowledge of reasoning and the accuracy of the generated CoT.
  arXiv  Detail & Related papers  (2024-02-17T05:22:56Z)
• Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning [14.569770617709073]
  We present a detailed analysis of which design choices cause instabilities and inconsistencies in task predictions. We show how spurious correlations between input distributions and labels form only a minor problem for prompted models. We statistically analyse the results to show which factors are the most influential, interactive or stable.
  arXiv  Detail & Related papers  (2023-10-20T13:25:24Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)
• Measuring Faithfulness in Chain-of-Thought Reasoning [19.074147845029355]
  Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question. It is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question) We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT.
  arXiv  Detail & Related papers  (2023-07-17T01:08:39Z)
• Question Decomposition Improves the Faithfulness of Model-Generated Reasoning [23.34325378824462]
  Large language models (LLMs) are difficult to verify the correctness and safety of their behavior. One approach is to prompt LLMs to externalize their reasoning, by having them generate step-by-step reasoning as they answer a question. This approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT.
  arXiv  Detail & Related papers  (2023-07-17T00:54:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.