Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness

• URL: http://arxiv.org/abs/2605.27879v1
• Date: Wed, 27 May 2026 02:56:30 GMT
• Title: Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness
• Abstract summary: We propose Faithful Agentic XAI, a framework that improves explanation faithfulness through explicit verification.<n>FAX decomposes draft explanations into claims and cross-checks them against inherently faithful tools.<n>On CRAFTER-XAI-Bench, FAX improves simulation faithfulness from 0.20 for the strongest baseline to 0.46 while maintaining high informativeness, relevance, and fluency.
• Score: 29.380265903481874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Explainable AI (XAI) helps users interpret model behavior and identify potential faults. Agentic XAI systems use Large Language Models (LLMs) to make explanations more accessible through natural-language interaction, but they can also produce plausible yet unfaithful explanations. This risk arises because unreliable XAI outputs for complex models can be amplified by LLMs and mislead users. We propose Faithful Agentic XAI (FAX), a framework that improves explanation faithfulness through explicit verification. FAX decomposes draft explanations into claims and cross-checks them against inherently faithful tools, filtering unsupported or contradictory claims before final generation. We also introduce CRAFTER-XAI-Bench, an open-world reinforcement learning benchmark with complex policies, diverse goals, and challenging scenarios for assessing model-specific faithfulness. On CRAFTER-XAI-Bench, FAX improves simulation faithfulness from 0.20 for the strongest baseline to 0.46 while maintaining high informativeness, relevance, and fluency. On three tabular benchmarks, FAX performs competitively with prior Agentic XAI baselines, but our analysis shows that these settings can conflate task accuracy with model-specific faithfulness. These findings show that explicit verification is essential for faithful Agentic XAI and that that faithfulness benchmarks must be designed to test explanations against the behavior of the target model itself.

Related papers

• From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents [69.07911499189129]
  We introduce KAPRO, a framework that evaluates cognitive-behavioral alignment by decoupling an agent's metacognitive judgment (Knowing) from its spontaneous execution (Acting)<n>Experiments show that self-awareness capability is strongly correlated with task success but degrades sharply in internal-capability settings.<n> proprietary and reasoning-oriented models demonstrate more reliable cognitive gating.
  arXiv  Detail & Related papers  (2026-06-09T17:17:08Z)
• A Two-Stage LLM Framework for Accessible and Verified XAI Explanations [0.0]
  Large Language Models (LLMs) are increasingly used to translate the technical outputs of eXplainable Artificial Intelligence (XAI) methods into accessible natural-language explanations.<n>This paper proposes a Two-Stage LLM Meta-Verification Framework that consists of (i) an Explainer LLM that converts raw XAI outputs into natural-language narratives, (ii) a Verifier LLM that assesses them in terms of faithfulness, coherence, completeness, and risk, and (iii) an iterative refeed mechanism that uses the Verifier's feedback to refine and improve them.
  arXiv  Detail & Related papers  (2026-04-14T10:15:57Z)
• Beyond single-model XAI: aggregating multi-model explanations for enhanced trustworthiness [43.25173443756643]
  This paper investigates the role of robustness through the usage of a feature importance aggregation derived from multiple models.<n>Preliminary results showcase the potential in increasing the trustworthiness of the application, while leveraging multiple model's predictive power.
  arXiv  Detail & Related papers  (2025-10-13T08:55:45Z)
• ThinkFake: Reasoning in Multimodal Large Language Models for AI-Generated Image Detection [51.93101033997245]
  Increasing realism of AI-generated images has raised serious concerns about misinformation and privacy violations.<n>We propose ThinkFake, a novel reasoning-based and generalizable framework for AI-generated image detection.<n>We show that ThinkFake outperforms state-of-the-art methods on the GenImage benchmark and demonstrates strong zero-shot generalization on the challenging LOKI benchmark.
  arXiv  Detail & Related papers  (2025-09-24T07:34:09Z)
• D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models [62.83226685925107]
  Deceptive Reasoning Exposure Suite (D-REX) is a novel dataset designed to evaluate the discrepancy between a model's internal reasoning process and its final output.<n>Each sample in D-REX contains the adversarial system prompt, an end-user's test query, the model's seemingly innocuous response, and, crucially, the model's internal chain-of-thought.<n>We demonstrate that D-REX presents a significant challenge for existing models and safety mechanisms.
  arXiv  Detail & Related papers  (2025-09-22T15:59:40Z)
• ForenX: Towards Explainable AI-Generated Image Detection with Multimodal Large Language Models [82.04858317800097]
  We present ForenX, a novel method that not only identifies the authenticity of images but also provides explanations that resonate with human thoughts.<n>ForenX employs the powerful multimodal large language models (MLLMs) to analyze and interpret forensic cues.<n>We introduce ForgReason, a dataset dedicated to descriptions of forgery evidences in AI-generated images.
  arXiv  Detail & Related papers  (2025-08-02T15:21:26Z)
• Explainable AI-Based Interface System for Weather Forecasting Model [21.801445160287532]
  This study defines three requirements for explanations of black-box models in meteorology through user studies.<n> Appropriate XAI methods are mapped to each requirement, and the generated explanations are tested quantitatively and qualitatively.<n>Results indicate that the explanations increase decision utility and user trust.
  arXiv  Detail & Related papers  (2025-04-01T13:52:34Z)
• Extending XReason: Formal Explanations for Adversarial Detection [0.0]
  We extend the XReason tool to support LightGBM models as well as class-level explanations.<n>We also implement a mechanism to generate and detect adversarial examples in XReason.<n>We evaluate the efficiency and accuracy of our approach on the CICIDS-2017 dataset.
  arXiv  Detail & Related papers  (2024-12-31T16:45:03Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• X Hacking: The Threat of Misguided AutoML [16.80633036922072]
  This paper introduces the concept of X-hacking, a form of p-hacking applied to XAI metrics such as SHAP values.<n>We show how easily an automated machine learning pipeline can be adapted to exploit model multiplicity at scale.<n>We show the vulnerability of a dataset to X-hacking can be determined by information redundancy among features.
  arXiv  Detail & Related papers  (2024-01-16T17:21:33Z)
• DeforestVis: Behavior Analysis of Machine Learning Models with Surrogate Decision Stumps [46.58231605323107]
  We propose DeforestVis, a visual analytics tool that offers summarization of the behaviour of complex ML models. DeforestVis helps users to explore the complexity versus fidelity trade-off by incrementally generating more stumps. We show the applicability and usefulness of DeforestVis with two use cases and expert interviews with data analysts and model developers.
  arXiv  Detail & Related papers  (2023-03-31T21:17:15Z)
• Counterfactual Explanations as Interventions in Latent Space [62.997667081978825]
  Counterfactual explanations aim to provide to end users a set of features that need to be changed in order to achieve a desired outcome. Current approaches rarely take into account the feasibility of actions needed to achieve the proposed explanations. We present Counterfactual Explanations as Interventions in Latent Space (CEILS), a methodology to generate counterfactual explanations.
  arXiv  Detail & Related papers  (2021-06-14T20:48:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.