When concept-based XAI is imprecise: Do people distinguish between generalisations and misrepresentations?

• URL: http://arxiv.org/abs/2506.17936v1
• Date: Sun, 22 Jun 2025 08:07:02 GMT
• Title: When concept-based XAI is imprecise: Do people distinguish between generalisations and misrepresentations?
• Authors: Romy Müller,
• Abstract summary: Concept-based explainable artificial intelligence (C-XAI) can help reveal the inner representations of AI models.<n>It may desirable for C-XAI concepts to show some variability.<n>It is unclear whether people recognise and appreciate such generalisations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Concept-based explainable artificial intelligence (C-XAI) can help reveal the inner representations of AI models. Understanding these representations is particularly important in complex tasks like safety evaluation. Such tasks rely on high-level semantic information (e.g., about actions) to make decisions about abstract categories (e.g., whether a situation is dangerous). In this context, it may desirable for C-XAI concepts to show some variability, suggesting that the AI is capable of generalising beyond the concrete details of a situation. However, it is unclear whether people recognise and appreciate such generalisations and can distinguish them from other, less desirable forms of imprecision. This was investigated in an experimental railway safety scenario. Participants evaluated the performance of a simulated AI that evaluated whether traffic scenes involving people were dangerous. To explain these decisions, the AI provided concepts in the form of similar image snippets. These concepts differed in their match with the classified image, either regarding a highly relevant feature (i.e., relation to tracks) or a less relevant feature (i.e., actions). Contrary to the hypotheses, concepts that generalised over less relevant features led to ratings that were lower than for precisely matching concepts and comparable to concepts that systematically misrepresented these features. Conversely, participants were highly sensitive to imprecisions in relevant features. These findings cast doubts on whether people spontaneously recognise generalisations. Accordingly, they might not be able to infer from C-XAI concepts whether AI models have gained a deeper understanding of complex situations.

Related papers

• Exploring the Impact of Explainable AI and Cognitive Capabilities on Users' Decisions [1.1049608786515839]
  Personality traits like the Need for Cognition (NFC) can lead to different decision-making outcomes among low and high NFC individuals.<n>We investigated how presenting AI information affects accuracy, reliance on AI, and cognitive load in a loan application scenario.<n>We found no significant differences between low and high NFC groups in accuracy or cognitive load, raising questions about the role of personality traits in AI-assisted decision-making.
  arXiv  Detail & Related papers  (2025-05-02T11:30:53Z)
• General Scales Unlock AI Evaluation with Explanatory and Predictive Power [57.7995945974989]
  benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems.<n>We introduce general scales for AI evaluation that can explain what common AI benchmarks really measure.<n>Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate.
  arXiv  Detail & Related papers  (2025-03-09T01:13:56Z)
• Hacking a surrogate model approach to XAI [49.1574468325115]
  We show that even if a discriminated subgroup does not get a positive decision from the black box ADM system, the corresponding question of group membership can be pushed down onto a level as low as wanted. Our approach can be generalized easily to other surrogate models.
  arXiv  Detail & Related papers  (2024-06-24T13:18:02Z)
• Relevant Irrelevance: Generating Alterfactual Explanations for Image Classifiers [11.200613814162185]
  In this paper, we demonstrate the feasibility of alterfactual explanations for black box image classifiers. We show for the first time that it is possible to apply this idea to black box models based on neural networks.
  arXiv  Detail & Related papers  (2024-05-08T11:03:22Z)
• Evaluating the Utility of Model Explanations for Model Development [54.23538543168767]
  We evaluate whether explanations can improve human decision-making in practical scenarios of machine learning model development. To our surprise, we did not find evidence of significant improvement on tasks when users were provided with any of the saliency maps. These findings suggest caution regarding the usefulness and potential for misunderstanding in saliency-based explanations.
  arXiv  Detail & Related papers  (2023-12-10T23:13:23Z)
• Explaining Explainability: Towards Deeper Actionable Insights into Deep Learning through Second-order Explainability [70.60433013657693]
  Second-order explainable AI (SOXAI) was recently proposed to extend explainable AI (XAI) from the instance level to the dataset level. We demonstrate for the first time, via example classification and segmentation cases, that eliminating irrelevant concepts from the training set based on actionable insights from SOXAI can enhance a model's performance.
  arXiv  Detail & Related papers  (2023-06-14T23:24:01Z)
• Disagreement amongst counterfactual explanations: How transparency can be deceptive [0.0]
  Counterfactual explanations are increasingly used as Explainable Artificial Intelligence technique. Not every algorithm creates uniform explanations for the same instance. Ethical issues arise when malicious agents use this diversity to fairwash an unfair machine learning model.
  arXiv  Detail & Related papers  (2023-04-25T09:15:37Z)
• Seamful XAI: Operationalizing Seamful Design in Explainable AI [59.89011292395202]
  Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps. We propose that seamful design can foster AI explainability by revealing sociotechnical and infrastructural mismatches. We explore this process with 43 AI practitioners and real end-users.
  arXiv  Detail & Related papers  (2022-11-12T21:54:05Z)
• Alterfactual Explanations -- The Relevance of Irrelevance for Explaining AI Systems [0.9542023122304099]
  We argue that in order to fully understand a decision, not only knowledge about relevant features is needed, but that the awareness of irrelevant information also highly contributes to the creation of a user's mental model of an AI system. Our approach, which we call Alterfactual Explanations, is based on showing an alternative reality where irrelevant features of an AI's input are altered. We show that alterfactual explanations are suited to convey an understanding of different aspects of the AI's reasoning than established counterfactual explanation methods.
  arXiv  Detail & Related papers  (2022-07-19T16:20:37Z)
• Conceptual Modeling and Artificial Intelligence: Mutual Benefits from Complementary Worlds [0.0]
  We are interested in tackling the intersection of the two, thus far, mostly isolated approached disciplines of CM and AI. The workshop embraces the assumption, that manifold mutual benefits can be realized by i) investigating what Conceptual Modeling (CM) can contribute to AI, and ii) the other way around.
  arXiv  Detail & Related papers  (2021-10-16T18:42:09Z)
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations [61.49776160925216]
  We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design.
  arXiv  Detail & Related papers  (2021-07-28T17:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.