What LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs

• URL: http://arxiv.org/abs/2606.28615v1
• Date: Fri, 26 Jun 2026 21:14:56 GMT
• Title: What LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs
• Abstract summary: Large language models (LLMs) are increasingly deployed in high-stakes domains.<n>It remains unclear whether these explanations are sufficient to explain the model's output-generating process.<n>We introduce an information-theoretic metric, SCSuff, that enables evaluation of free-text explanations.
• Score: 30.540142439631072
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed in high-stakes domains, where free-text explanations such as chain-of-thought and post-hoc rationales are used to justify model outputs. Yet it remains unclear whether these explanations are sufficient, i.e., if they contain enough information to explain the model's output-generating process. We generalize classical sufficiency from feature attributions to arbitrary explanations and prove that explanation sufficiency can change depending on the input distribution, which must be explicitly defined for LLM explanations. We propose using the LLM itself to generate alternative inputs conditioned on an explanation, capturing its beliefs about possible inputs. We formalize self-consistent sufficiency as a goal for free-text explanations and introduce an information-theoretic metric, SCSuff, that enables evaluation of free-text explanations without relying on predefined biases or shortcuts. Our experiments show that SCSuff agrees with targeted perturbation tests where applicable and demonstrate that explanation sufficiency can vary with the input distribution. We find LLM explanations are generally insufficient and weakly correlated with model size, accuracy, or output entropy. Analysis of final-token hidden states shows that top and bottom SCSuff scores can be predicted from internal representations, suggesting that SCSuff can guide detection and improvement of sufficient LLM explanations. The code for this paper is available at https://github.com/rajesh-lab/self-consistent-sufficiency .

Related papers

• Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers [0.47248250311484113]
  In-context learning (ICL) enables multimodal large language models (MLLMs) to classify images from a few labelled examples.<n>We evaluate the concept-based explainability of frozen MLLMs under few-shot ICL using five conditions of increasing formal rigour.
  arXiv  Detail & Related papers  (2026-05-27T09:32:34Z)
• Faithfulness Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance [57.17102098930037]
  Large language models (LLMs) achieve strong performance and have revolutionized NLP.<n>Their lack of explainability keeps them treated as black boxes, limiting their use in domains that demand transparency and trust.<n>We introduce a training-free method that enhances faithfulness by guiding explanation generation through attention-level interventions.
  arXiv  Detail & Related papers  (2026-04-15T18:32:32Z)
• From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks [49.57538588967748]
  Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks.<n>We uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma.<n>We show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph.
  arXiv  Detail & Related papers  (2026-04-01T11:40:12Z)
• Training Language Models to Explain Their Own Computations [73.8562596518326]
  We study the extent to which LMs' privileged access to their own internals can be leveraged to produce new techniques for explaining their behavior.<n>Using existing interpretability techniques as a source of ground truth, we fine-tune LMs to generate natural language descriptions of (1) the information encoded by LM features, (2) the causal structure of LMs' internal activations, and (3) the influence of specific input tokens on LM outputs.
  arXiv  Detail & Related papers  (2025-11-11T18:57:14Z)
• Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations [0.8949668577519213]
  Large language models (LLMs) are capable of generating plausible explanations of how they arrived at an answer to a question.<n>These explanations can misrepresent the model's "reasoning" process, i.e., they can be unfaithful.<n>We introduce a new approach for measuring the faithfulness of LLM explanations.
  arXiv  Detail & Related papers  (2025-04-19T02:51:20Z)
• Are We Merely Justifying Results ex Post Facto? Quantifying Explanatory Inversion in Post-Hoc Model Explanations [87.68633031231924]
  Post-hoc explanation methods provide interpretation by attributing predictions to input features.<n>Do these explanations unintentionally reverse the natural relationship between inputs and outputs?<n>We propose Inversion Quantification (IQ), a framework that quantifies the degree to which explanations rely on outputs and deviate from faithful input-output relationships.
  arXiv  Detail & Related papers  (2025-04-11T19:00:12Z)
• Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoring [18.837335987273256]
  Large language models (LLMs) are becoming increasingly capable, but the mechanisms of their thinking and decision-making process remain unclear.<n>We propose a novel method TELLME, improving the transparency of LLMs and helping monitors identify unsuitable and sensitive behaviors.
  arXiv  Detail & Related papers  (2025-02-07T13:25:33Z)
• Explanation sensitivity to the randomness of large language models: the case of journalistic text classification [6.240875403446504]
  We study the effect of random elements in the training of large language models on the explainability of their predictions. Using a fine-tuned CamemBERT model and an explanation method based on relevance propagation, we find that training with different random seeds produces models with similar accuracy but variable explanations.
  arXiv  Detail & Related papers  (2024-10-07T14:39:45Z)
• Evaluating Evidence Attribution in Generated Fact Checking Explanations [48.776087871960584]
  We introduce a novel evaluation protocol -- citation masking and recovery.<n>We implement our protocol using both human annotators and automatic annotators.<n>Experiments reveal that the best-performing LLMs still generate explanations with inaccurate attributions.
  arXiv  Detail & Related papers  (2024-06-18T14:13:13Z)
• A Hypothesis-Driven Framework for the Analysis of Self-Rationalising Models [0.8702432681310401]
  We use a Bayesian network to implement a hypothesis about how a task is solved. The resulting models do not exhibit a strong similarity to GPT-3.5. We discuss the implications of this as well as the framework's potential to approximate LLM decisions better in future work.
  arXiv  Detail & Related papers  (2024-02-07T12:26:12Z)
• FaithLM: Towards Faithful Explanations for Large Language Models [60.45183469474916]
  We introduce FaithLM, a model-agnostic framework that evaluates and improves the faithfulness of large language models.<n>We show that FaithLM consistently increases faithfulness and produces explanations more aligned with human rationales than strong self-explanation baselines.
  arXiv  Detail & Related papers  (2024-02-07T09:09:14Z)
• Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? [86.60613602337246]
  We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations. LAS measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage.
  arXiv  Detail & Related papers  (2020-10-08T16:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.