An Empirical Study of Counterfactual Self-Explanations in LLMs

• URL: http://arxiv.org/abs/2609.17119v1
• Date: Tue, 15 Sep 2026 12:47:22 GMT
• Title: An Empirical Study of Counterfactual Self-Explanations in LLMs
• Abstract summary: Large language models can easily generate explanations for their own outputs, but such self-explanations are not necessarily faithful to the model's behavior.<n>We study this issue through counterfactual self-explanations, where a model minimally edits an input so that its own prediction changes.<n>Our results show that model scale is the strongest determinant of explanation quality.
• Score: 13.093234754814851
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models can easily generate explanations for their own outputs, but such self-explanations are not necessarily faithful to the model's behavior. We study this issue through counterfactual self-explanations, where a model minimally edits an input so that its own prediction changes. Across sentiment analysis and natural language inference, we evaluate ten instruction-tuned models from the LLaMA-3 and Qwen-2.5 families, measuring faithfulness, minimality, and alignment with human-annotated rationales. Our results show that model scale is the strongest determinant of explanation quality: larger models are substantially more likely to generate counterfactuals that flip their own predictions and target decision-relevant evidence. In contrast, the rationale-guided condition produces edit-minimal counterfactuals that are also more human-aligned. However, it does not consistently improve faithfulness. Overall, counterfactual self-explanations can provide useful behavioral evidence about model decisions, but their reliability depends strongly on model capacity and should be empirically validated rather than assumed.

Related papers

• A Removal Based Approach to Improve LLM Faithfulness at Test-Time [2.6612030624044194]
  Large language models (LLMs) are increasingly used for consequential decisions.<n>These explanations can be unfaithful, failing to reflect the actual reasoning underlying the model's decisions.<n>We introduce a test-time approach that directly targets incompleteness.
  arXiv  Detail & Related papers  (2026-09-03T18:09:26Z)
• Amplified Does Not Mean Predictive: Reasoning Behaviors in Thinking Models [56.88565331713778]
  reasoning-oriented training can make traces look more deliberative without amplifying the behaviors most tied to model correctness.<n>We quantify this mismatch with Behavioral Lift, a metric that measures how much correctness changes when a behavior is present versus absent in a model's reasoning trace.<n>We find evidence for an Amplification-Lift Gap, in which thinking models strongly amplify self-awareness, hypothesis testing, and uncertainty acknowledgment.
  arXiv  Detail & Related papers  (2026-08-13T20:37:59Z)
• A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior [11.616524876789624]
  LLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the model's true reasoning process is poorly understood.<n>We introduce Normalized Simulata Gainbility (NSG), a metric based on the idea that a faithful explanation should allow an observer to learn a model's decision-making criteria.<n>We find self-explanations substantially improve prediction of model behavior (11-37% NSG)
  arXiv  Detail & Related papers  (2026-02-02T18:54:51Z)
• Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations [1.8772057593980798]
  Large Language Models (LLMs) can produce verbalized self-explanations.<n>We evaluate how well humans and LLM judges can predict a model's answers to counterfactual follow-up questions.
  arXiv  Detail & Related papers  (2026-01-07T10:13:26Z)
• Language Models can perform Single-Utterance Self-Correction of Perturbed Reasoning [4.768151813962547]
  Large Language Models (LLMs) have demonstrated impressive mathematical reasoning capabilities.<n>Their performance remains brittle to minor variations in problem description and prompting strategy.<n>To better understand self-correction capabilities of recent models, we conduct experiments measuring models' ability to self-correct synthetics.
  arXiv  Detail & Related papers  (2025-06-18T21:35:44Z)
• Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability [70.4107059502882]
  Training language models with rationales augmentation has been shown to be beneficial in many existing works.<n>We conduct comprehensive investigations to thoroughly inspect the impact of rationales on model performance.
  arXiv  Detail & Related papers  (2025-05-30T02:39:37Z)
• Thinking Out Loud: Do Reasoning Models Know When They're Right? [25.683190106676893]
  Large reasoning models (LRMs) have recently demonstrated impressive capabilities in complex reasoning tasks.<n>We investigate how LRMs interact with other model behaviors by analyzing verbalized confidence.<n>We find that reasoning models may possess a diminished awareness of their own knowledge boundaries.
  arXiv  Detail & Related papers  (2025-04-09T03:58:19Z)
• Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learning [76.98542249776257]
  Large-scale language models often face the challenge of "hallucination" We introduce an uncertainty-aware in-context learning framework to empower the model to enhance or reject its output in response to uncertainty.
  arXiv  Detail & Related papers  (2023-10-07T12:06:53Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• To what extent do human explanations of model behavior align with actual model behavior? [91.67905128825402]
  We investigated the extent to which human-generated explanations of models' inference decisions align with how models actually make these decisions. We defined two alignment metrics that quantify how well natural language human explanations align with model sensitivity to input words. We find that a model's alignment with human explanations is not predicted by the model's accuracy on NLI.
  arXiv  Detail & Related papers  (2020-12-24T17:40:06Z)
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction [49.254162397086006]
  We study explanations based on visual saliency in an image-based age prediction task. We find that presenting model predictions improves human accuracy. However, explanations of various kinds fail to significantly alter human accuracy or trust in the model.
  arXiv  Detail & Related papers  (2020-07-23T20:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.