Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding

• URL: http://arxiv.org/abs/2603.25251v1
• Date: Thu, 26 Mar 2026 09:55:10 GMT
• Title: Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding
• Abstract summary: Higher correctness is assumed to produce better human understanding, but this link has not been tested experimentally with controlled levels.<n>We show that not all differences in functional correctness translate to differences in human understanding, underscoring the need to validate functional metrics against human outcomes.
• Score: 5.619746163828999
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Explainable AI (XAI) methods are commonly evaluated with functional metrics such as correctness, which computationally estimate how accurately an explanation reflects the model's reasoning. Higher correctness is assumed to produce better human understanding, but this link has not been tested experimentally with controlled levels. We conducted a user study (N=200) that manipulated explanation correctness at four levels (100%, 85%, 70%, 55%) in a time series classification task where participants could not rely on domain knowledge or visual intuition and instead predicted the AI's decisions based on explanations (forward simulation). Correctness affected understanding, but not at every level: performance dropped at 70% and 55% correctness relative to fully correct explanations, while further degradation below 70% produced no additional loss. Rather than shifting performance uniformly, lower correctness decreased the proportion of participants who learned the decision pattern. At the same time, even fully correct explanations did not guarantee understanding, as only a subset of participants achieved high accuracy. Exploratory analyses showed that self-reported ratings correlated with demonstrated performance only when explanations were fully correct and participants had learned the pattern. These findings show that not all differences in functional correctness translate to differences in human understanding, underscoring the need to validate functional metrics against human outcomes.

Related papers

• Catching The Correct Answer Trap: Characterising AI Tutor Blind Spots When Analysing Student Reasoning [2.6352293843915606]
  We study a failure mode we call the correct answer trap (CAT): models under-detect misconceptions when students reach a correct answer via flawed reasoning.<n>Analysing real student responses from the Eedi mathematics platform, we show that 71% of these failures concentrate in just two question types.
  arXiv  Detail & Related papers  (2026-04-20T10:53:48Z)
• Pando: Do Interpretability Methods Work When Models Won't Explain Themselves? [53.07826484214082]
  We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis.<n>Pando predicts held-out model decisions from $10$ labeled query-response pairs.<n>When explanations are faithful, black-box elicitation matches or exceeds all white-box methods.
  arXiv  Detail & Related papers  (2026-04-13T06:42:24Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models [108.26461635308796]
  We introduce Rationale Consistency, a fine-grained metric that quantifies the alignment between the model's reasoning process and human judgment.<n>Our evaluation of frontier models reveals that rationale consistency effectively discriminates among state-of-the-art models.<n>We introduce a hybrid signal that combines rationale consistency with outcome accuracy for GenRM training.
  arXiv  Detail & Related papers  (2026-02-04T15:24:52Z)
• Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course [0.7515394929245942]
  This paper is a student-led field experiment embedded as a midterm project in UNIV 182 (AI4All) at George Mason University.<n>Students designed their own reasoning tasks, ran them on widely used consumer chatbots representative of current capabilities, and evaluated both (i) answer correctness and (ii) the validity of the reasoning.<n>OpenAI GPT-5 and Claude 4.5 achieved the highest mean answer accuracy (86.2% and 83.8%), followed by Grok 4 (82.5%) and Perplexity (73.1%)
  arXiv  Detail & Related papers  (2025-12-28T22:51:25Z)
• A suite of LMs comprehend puzzle statements as well as humans [13.386647125288516]
  We report a preregistered study comparing human responses in two conditions: one allowed rereading, and one that restricted rereading.<n>Human accuracy dropped significantly when rereading was restricted, falling below that of Falcon-180B-Chat and GPT-4.<n>Results suggest shared pragmatic sensitivities rather than model-specific deficits.
  arXiv  Detail & Related papers  (2025-05-13T22:18:51Z)
• Predicting Satisfaction of Counterfactual Explanations from Human Ratings of Explanatory Qualities [0.873811641236639]
  We analyze a dataset of counterfactual explanations that were evaluated by 206 human participants.<n>We find that feasibility and trust stand out as the strongest predictors of user satisfaction.<n>Other metrics explain 58% of the variance, highlighting the importance of additional explanatory qualities.
  arXiv  Detail & Related papers  (2025-04-07T11:09:25Z)
• LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback [71.95402654982095]
  We propose Math-Minos, a natural language feedback-enhanced verifier. Our experiments reveal that a small set of natural language feedback can significantly boost the performance of the verifier.
  arXiv  Detail & Related papers  (2024-06-20T06:42:27Z)
• Pre-training and Diagnosing Knowledge Base Completion Models [58.07183284468881]
  We introduce and analyze an approach to knowledge transfer from one collection of facts to another without the need for entity or relation matching. The main contribution is a method that can make use of large-scale pre-training on facts, which were collected from unstructured text. To understand the obtained pre-trained models better, we then introduce a novel dataset for the analysis of pre-trained models for Open Knowledge Base Completion.
  arXiv  Detail & Related papers  (2024-01-27T15:20:43Z)
• Don't Explain Noise: Robust Counterfactuals for Randomized Ensembles [50.81061839052459]
  We formalize the generation of robust counterfactual explanations as a probabilistic problem. We show the link between the robustness of ensemble models and the robustness of base learners. Our method achieves high robustness with only a small increase in the distance from counterfactual explanations to their initial observations.
  arXiv  Detail & Related papers  (2022-05-27T17:28:54Z)
• Can Active Learning Preemptively Mitigate Fairness Issues? [66.84854430781097]
  dataset bias is one of the prevailing causes of unfairness in machine learning. We study whether models trained with uncertainty-based ALs are fairer in their decisions with respect to a protected class. We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD.
  arXiv  Detail & Related papers  (2021-04-14T14:20:22Z)
• SCOUT: Self-aware Discriminant Counterfactual Explanations [78.79534272979305]
  The problem of counterfactual visual explanations is considered. A new family of discriminant explanations is introduced. The resulting counterfactual explanations are optimization free and thus much faster than previous methods.
  arXiv  Detail & Related papers  (2020-04-16T17:05:49Z)
• How human judgment impairs automated deception detection performance [0.5660207256468972]
  We tested whether a combination of supervised machine learning and human judgment could improve deception detection accuracy. Human involvement through hybrid-overrule decisions brought the accuracy back to the chance level. The decision-making strategies of humans suggest that the truth bias - the tendency to assume the other is telling the truth - could explain the detrimental effect.
  arXiv  Detail & Related papers  (2020-03-30T10:06:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.