Beyond SHAP and Anchors: A large-scale experiment on how developers struggle to design meaningful end-user explanations

• URL: http://arxiv.org/abs/2503.15512v3
• Date: Thu, 25 Sep 2025 08:48:45 GMT
• Title: Beyond SHAP and Anchors: A large-scale experiment on how developers struggle to design meaningful end-user explanations
• Authors: Zahra Abba Omar, Nadia Nahar, Jacob Tjaden, Inès M. Gilles, Fikir Mekonnen, Erica Okeh, Jane Hsieh, Christian Kästner, Alka Menon,
• Abstract summary: Modern machine learning produces models that are impossible for users or developers to fully understand.<n> Transparency and explainability methods aim to provide some help in understanding models.<n>Emerging guidelines and regulations set goals but may not provide effective actionable guidance to developers.
• Score: 11.20554074076788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern machine learning produces models that are impossible for users or developers to fully understand -- raising concerns about trust, oversight, safety, and human dignity when they are integrated into software products. Transparency and explainability methods aim to provide some help in understanding models, but it remains challenging for developers to design explanations that are understandable to target users and effective for their purpose. Emerging guidelines and regulations set goals but may not provide effective actionable guidance to developers. In a large-scale experiment with 124 participants, we explored how developers approach providing end-user explanations, including what challenges they face, and to what extent specific policies can guide their actions. We investigated whether and how specific forms of policy guidance help developers design explanations and provide evidence for policy compliance for an ML-powered screening tool for diabetic retinopathy. Participants across the board struggled to produce quality explanations and comply with the provided policies. Contrary to our expectations, we found that the nature and specificity of policy guidance had little effect. We posit that participant noncompliance is in part due to a failure to imagine and anticipate the needs of non-technical stakeholders. Drawing on cognitive process theory and the sociological imagination to contextualize participants' failure, we recommend educational interventions.

Related papers

• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.<n>Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes.<n>We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice [186.055899073629]
  Unlearning is often invoked as a solution for removing the effects of targeted information from a generative-AI model.<n>Unlearning is also proposed as a way to prevent a model from generating targeted types of information in its outputs.<n>Both of these goals--the targeted removal of information from a model and the targeted suppression of information from a model's outputs--present various technical and substantive challenges.
  arXiv  Detail & Related papers  (2024-12-09T20:18:43Z)
• From Critique to Clarity: A Pathway to Faithful and Personalized Code Explanations with Large Language Models [3.5176327869934747]
  This paper presents an innovative approach to generate faithful and personalized code explanations.<n>Our methodology integrates prompt enhancement, self-consuming mechanisms, personalized content customization, and interaction with external tools.<n>Our findings suggest that this approach significantly improves the quality and relevance of code explanations.
  arXiv  Detail & Related papers  (2024-12-08T09:02:04Z)
• Evaluating the Utility of Model Explanations for Model Development [54.23538543168767]
  We evaluate whether explanations can improve human decision-making in practical scenarios of machine learning model development. To our surprise, we did not find evidence of significant improvement on tasks when users were provided with any of the saliency maps. These findings suggest caution regarding the usefulness and potential for misunderstanding in saliency-based explanations.
  arXiv  Detail & Related papers  (2023-12-10T23:13:23Z)
• Optimising Human-AI Collaboration by Learning Convincing Explanations [62.81395661556852]
  We propose a method for a collaborative system that remains safe by having a human making decisions. Ardent enables efficient and effective decision-making by adapting to individual preferences for explanations.
  arXiv  Detail & Related papers  (2023-11-13T16:00:16Z)
• Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning [72.80902932543474]
  Understanding human behavior from observed data is critical for transparency and accountability in decision-making. Consider real-world settings such as healthcare, in which modeling a decision-maker's policy is challenging. We propose a data-driven representation of decision-making behavior that inheres transparency by design, accommodates partial observability, and operates completely offline.
  arXiv  Detail & Related papers  (2023-10-28T13:06:14Z)
• Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection [70.28425745910711]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following. This capability brings with it the risk of prompt injection attacks. We evaluate the robustness of instruction-following LLMs against such attacks.
  arXiv  Detail & Related papers  (2023-08-17T06:21:50Z)
• Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations [11.667611038005552]
  We take a step back from sophisticated predictive algorithms and look into explainability of simple decision-making models. We aim to assess how people perceive comprehensibility of their different representations. This allows us to capture how diverse stakeholders judge intelligibility of fundamental concepts that more elaborate artificial intelligence explanations are built from.
  arXiv  Detail & Related papers  (2023-03-02T03:15:35Z)
• Redefining Counterfactual Explanations for Reinforcement Learning: Overview, Challenges and Opportunities [2.0341936392563063]
  Most explanation methods for AI are focused on developers and expert users. Counterfactual explanations offer users advice on what can be changed in the input for the output of the black-box model to change. Counterfactuals are user-friendly and provide actionable advice for achieving the desired output from the AI system.
  arXiv  Detail & Related papers  (2022-10-21T09:50:53Z)
• Experiential Explanations for Reinforcement Learning [18.039927414134088]
  Reinforcement learning systems can be complex and non-interpretable.<n>We propose a technique Experiential Explanations to generate counterfactual explanations.
  arXiv  Detail & Related papers  (2022-10-10T14:27:53Z)
• Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies [79.60322329952453]
  We show how to develop interpretable representations of how agents make decisions. By understanding the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
  arXiv  Detail & Related papers  (2022-03-14T17:40:42Z)
• Towards a Responsible AI Development Lifecycle: Lessons From Information Security [0.0]
  We propose a framework for responsibly developing artificial intelligence systems. In particular, we propose leveraging the concepts of threat modeling, design review, penetration testing, and incident response.
  arXiv  Detail & Related papers  (2022-03-06T13:03:58Z)
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty [66.17147341354577]
  We argue for considering a complementary form of transparency by estimating and communicating the uncertainty associated with model predictions. We describe how uncertainty can be used to mitigate model unfairness, augment decision-making, and build trustworthy systems. This work constitutes an interdisciplinary review drawn from literature spanning machine learning, visualization/HCI, design, decision-making, and fairness.
  arXiv  Detail & Related papers  (2020-11-15T17:26:14Z)
• Explainability Case Studies [2.2872132127037963]
  Explainability is one of the key ethical concepts in the design of AI systems. We present a set of case studies of a hypothetical AI-enabled product, which serves as a pedagogical tool to empower product designers, developers, students, and educators to develop a holistic explainability strategy for their own products.
  arXiv  Detail & Related papers  (2020-09-01T05:54:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.