Explainability Pitfalls: Beyond Dark Patterns in Explainable AI

• URL: http://arxiv.org/abs/2109.12480v1
• Date: Sun, 26 Sep 2021 03:05:50 GMT
• Title: Explainability Pitfalls: Beyond Dark Patterns in Explainable AI
• Abstract summary: We introduce explainability pitfalls (EPs), unanticipated negative downstream effects from AI explanations manifesting even when there is no intention to manipulate users. EPs are different from, yet related to, dark patterns, which are intentionally deceptive practices. We propose proactive and preventative strategies to address EPs at three interconnected levels.
• Score: 9.020313208419378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To make Explainable AI (XAI) systems trustworthy, understanding harmful effects is just as important as producing well-designed explanations. In this paper, we address an important yet unarticulated type of negative effect in XAI. We introduce explainability pitfalls(EPs), unanticipated negative downstream effects from AI explanations manifesting even when there is no intention to manipulate users. EPs are different from, yet related to, dark patterns, which are intentionally deceptive practices. We articulate the concept of EPs by demarcating it from dark patterns and highlighting the challenges arising from uncertainties around pitfalls. We situate and operationalize the concept using a case study that showcases how, despite best intentions, unsuspecting negative effects such as unwarranted trust in numerical explanations can emerge. We propose proactive and preventative strategies to address EPs at three interconnected levels: research, design, and organizational.

Related papers

• Warning labels shift perceptions of sycophantic AI, but not its influence [63.79516339973759]
  We test whether warning labels mitigate sycophancy's influence.<n>Labeling the system as sycophantic (...may agree with you and validate you even when you are wrong...'') does shift users' perceptions.<n>But it does not reliably reduce sycophancy's influence on users' self-perceived rightness or their willingness to repair the conflict.
  arXiv  Detail & Related papers  (2026-06-19T11:01:45Z)
• Position: Explainability Research Must Prioritize Foundations over Ad-hoc Methods [56.14010017381489]
  This paper argues that the machine learning community must pivot from ad-hoc XAI methods toward addressing foundational & structural challenges.<n>We conclude by outlining a practical checklist designed to shift XAI toward a more human-centered, action-oriented paradigm.
  arXiv  Detail & Related papers  (2026-06-18T21:46:20Z)
• Imagination Helps Visual Reasoning, But Not Yet in Latent Space [65.80396132375571]
  We investigate the validity of latent reasoning using Causal Mediation Analysis.<n>We show that latent tokens encode limited visual information and exhibit high similarity.<n>We propose a straightforward alternative named CapImagine, which teaches the model to explicitly imagine using text.
  arXiv  Detail & Related papers  (2026-02-26T08:56:23Z)
• From Narrow Unlearning to Emergent Misalignment: Causes, Consequences, and Containment in LLMs [51.800006486987435]
  We show that emergent misalignment can arise from narrow refusal unlearning in specific domains.<n>Our work shows that narrow domain unlearning can yield compliance responses for the targeted concept, however, it may also propagate EMA to unrelated domains.
  arXiv  Detail & Related papers  (2025-11-18T00:53:23Z)
• Beware of "Explanations" of AI [16.314859121110945]
  Understanding the decisions made and actions taken by increasingly complex AI system remains a key challenge. This has led to an expanding field of research in explainable artificial intelligence (XAI) The question of what constitutes a "good" explanation is dependent on the goals, stakeholders, and context.
  arXiv  Detail & Related papers  (2025-04-09T11:31:08Z)
• Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration [11.824688232910193]
  We run a study on AI-assisted decision-making in which humans were supported by XAI. Our findings reveal a misinformation effect when incorrect explanations accompany correct AI advice. This effect causes humans to infer flawed reasoning strategies, hindering task execution and demonstrating impaired procedural knowledge.
  arXiv  Detail & Related papers  (2024-09-19T14:34:20Z)
• Understanding the Impact of Negative Prompts: When and How Do They Take Effect? [92.53724347718173]
  This paper presents the first comprehensive study to uncover how and when negative prompts take effect. Our empirical analysis identifies two primary behaviors of negative prompts. Negative prompts can facilitate object inpainting with minimal alterations to the background via a simple adaptive algorithm.
  arXiv  Detail & Related papers  (2024-06-05T05:42:46Z)
• How Well Do Feature-Additive Explainers Explain Feature-Additive Predictors? [12.993027779814478]
  We ask the question: can popular feature-additive explainers (e.g., LIME, SHAP, SHAPR, MAPLE, and PDP) explain feature-additive predictors? Herein, we evaluate such explainers on ground truth that is analytically derived from the additive structure of a model. Our results suggest that all explainers eventually fail to correctly attribute the importance of features, especially when a decision-making process involves feature interactions.
  arXiv  Detail & Related papers  (2023-10-27T21:16:28Z)
• Seamful XAI: Operationalizing Seamful Design in Explainable AI [59.89011292395202]
  Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps. We propose that seamful design can foster AI explainability by revealing sociotechnical and infrastructural mismatches. We explore this process with 43 AI practitioners and real end-users.
  arXiv  Detail & Related papers  (2022-11-12T21:54:05Z)
• Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making [10.049226270783562]
  We study the effects of feature-based explanations on distributive fairness of AI-assisted decisions. Our findings show that explanations influence fairness perceptions, which, in turn, relate to humans' tendency to adhere to AI recommendations.
  arXiv  Detail & Related papers  (2022-09-23T19:10:59Z)
• Diagnosing AI Explanation Methods with Folk Concepts of Behavior [70.10183435379162]
  We consider "success" to depend not only on what information the explanation contains, but also on what information the human explainee understands from it. We use folk concepts of behavior as a framework of social attribution by the human explainee.
  arXiv  Detail & Related papers  (2022-01-27T00:19:41Z)
• Towards Relatable Explainable AI with the Perceptual Process [5.581885362337179]
  We argue that explanations must be more relatable to other concepts, hypotheticals, and associations. Inspired by cognitive psychology, we propose the XAI Perceptual Processing Framework and RexNet model for relatable explainable AI.
  arXiv  Detail & Related papers  (2021-12-28T05:48:53Z)
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations [61.49776160925216]
  We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design.
  arXiv  Detail & Related papers  (2021-07-28T17:32:04Z)
• Counterfactual Explanations as Interventions in Latent Space [62.997667081978825]
  Counterfactual explanations aim to provide to end users a set of features that need to be changed in order to achieve a desired outcome. Current approaches rarely take into account the feasibility of actions needed to achieve the proposed explanations. We present Counterfactual Explanations as Interventions in Latent Space (CEILS), a methodology to generate counterfactual explanations.
  arXiv  Detail & Related papers  (2021-06-14T20:48:48Z)
• Overcoming Failures of Imagination in AI Infused System Development and Deployment [71.9309995623067]
  NeurIPS 2020 requested that research paper submissions include impact statements on "potential nefarious uses and the consequences of failure" We argue that frameworks of harms must be context-aware and consider a wider range of potential stakeholders, system affordances, as well as viable proxies for assessing harms in the widest sense.
  arXiv  Detail & Related papers  (2020-11-26T18:09:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.