When AI Gets it Wrong: Reliability and Risk in AI-Assisted Medication Decision Systems

• URL: http://arxiv.org/abs/2604.01449v2
• Date: Fri, 03 Apr 2026 11:56:46 GMT
• Title: When AI Gets it Wrong: Reliability and Risk in AI-Assisted Medication Decision Systems
• Abstract summary: In high-risk domains such as medication management, even a single incorrect recommendation can result in severe patient harm.<n>This paper examines the reliability of AI-assisted medication systems by focusing on system failures and their potential clinical consequences.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Artificial intelligence (AI) systems are increasingly integrated into healthcare and pharmacy workflows, supporting tasks such as medication recommendations, dosage determination, and drug interaction detection. While these systems often demonstrate strong performance under standard evaluation metrics, their reliability in real-world decision-making remains insufficiently understood. In high-risk domains such as medication management, even a single incorrect recommendation can result in severe patient harm. This paper examines the reliability of AI-assisted medication systems by focusing on system failures and their potential clinical consequences. Rather than evaluating performance solely through aggregate metrics, this work shifts attention towards how errors occur and what happens when AI systems produce incorrect outputs. Through a series of controlled, simulated scenarios involving drug interactions and dosage decisions, we analyse different types of system failures, including missed interactions, incorrect risk flagging, and inappropriate dosage recommendations. The findings highlight that AI errors in medication-related contexts can lead to adverse drug reactions, ineffective treatment, or delayed care, particularly when systems are used without sufficient human oversight. Furthermore, the paper discusses the risks of over-reliance on AI recommendations and the challenges posed by limited transparency in decision-making processes. This work contributes a reliability-focused perspective on AI evaluation in healthcare, emphasising the importance of understanding failure behavior and real-world impact. It highlights the need to complement traditional performance metrics with risk-aware evaluation approaches, particularly in safety-critical domains such as pharmacy practice.

Related papers

• Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
  Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-17T10:26:37Z)
• Enhancing the Reliability of Medical AI through Expert-guided Uncertainty Modeling [35.159273667448176]
  unpredictability of AI errors poses a significant challenge, particularly in healthcare contexts.<n>We propose a novel approach that leverages disagreement in expert responses to generate targets for training machine learning models.<n>Our experiments demonstrate that incorporating expert knowledge can enhance uncertainty estimation quality by $9%$ to $50%$ depending on the task.
  arXiv  Detail & Related papers  (2026-04-02T11:01:31Z)
• Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI [45.21562889170875]
  We propose a systematic evaluation paradigm that characterizes agent trustworthiness over a scenario manifold.<n>The framework integrates four complementary components: static cognitive and policy analysis, (ii) interactive sandbox simulation, (iii) social-ethical alignment assessment, and (iv) a distribution-aware representative sampling engine.
  arXiv  Detail & Related papers  (2026-03-16T08:51:33Z)
• When Life Gives You AI, Will You Turn It Into A Market for Lemons? Understanding How Information Asymmetries About AI System Capabilities Affect Market Outcomes and Adoption [45.10829096284761]
  Complex AI systems can appear highly accurate while making costly errors or embedding hidden defects.<n>This paper provides the first experimental evidence on the role of information asymmetries and disclosure designs in shaping user adoption of AI systems.
  arXiv  Detail & Related papers  (2026-01-29T12:49:28Z)
• AI Epidemiology: achieving explainable AI through expert oversight patterns [0.0]
  AI Epidemiology is a framework for governing and explaining advanced AI systems.<n>It mirrors the way in which epidemiologists enable public health interventions through statistical evidence.
  arXiv  Detail & Related papers  (2025-12-15T11:29:05Z)
• Monitoring Deployed AI Systems in Health Care [8.240297189233528]
  Post-deployment monitoring of artificial intelligence (AI) systems in health care is essential to ensure their safety, quality, and sustained benefit.<n>This framework is organized around three complementary principles: system integrity, performance, and impact.<n>It offers a practical template and starting point for health systems seeking to ensure that AI deployments remain safe and effective over time.
  arXiv  Detail & Related papers  (2025-12-09T19:06:48Z)
• Perspectives on a Reliability Monitoring Framework for Agentic AI Systems [5.539407031861404]
  We derive the main reliability challenges of agentic AI systems during operation based on their characteristics.<n>We propose a two-layered reliability monitoring framework for agentic AI systems.
  arXiv  Detail & Related papers  (2025-11-12T10:19:17Z)
• MARIA: A Framework for Marginal Risk Assessment without Ground Truth in AI Systems [11.620099531890716]
  Before deploying an AI system to replace an existing process, it must be compared with the incumbent to ensure improvement without added risk.<n>Traditional evaluation relies on ground truth for both systems, but this is often unavailable due to delayed or unknowable outcomes.<n>We propose a marginal risk assessment framework, that avoids dependence on ground truth or absolute risk.
  arXiv  Detail & Related papers  (2025-10-31T04:18:20Z)
• AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons [62.374792825813394]
  This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability.<n>The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories.
  arXiv  Detail & Related papers  (2025-02-19T05:58:52Z)
• Model evaluation for extreme risks [46.53170857607407]
  Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why model evaluation is critical for addressing extreme risks.
  arXiv  Detail & Related papers  (2023-05-24T16:38:43Z)
• Validation and Transparency in AI systems for pharmacovigilance: a case study applied to the medical literature monitoring of adverse events [0.483420384410068]
  We present a case study on how to operationalize existing guidance for validated AI systems in pharmacovigilance. We describe an AI system designed with the goal of reducing effort to mitigate activities built in close collaboration with subject matter experts.
  arXiv  Detail & Related papers  (2021-12-21T21:02:24Z)
• Overcoming Failures of Imagination in AI Infused System Development and Deployment [71.9309995623067]
  NeurIPS 2020 requested that research paper submissions include impact statements on "potential nefarious uses and the consequences of failure" We argue that frameworks of harms must be context-aware and consider a wider range of potential stakeholders, system affordances, as well as viable proxies for assessing harms in the widest sense.
  arXiv  Detail & Related papers  (2020-11-26T18:09:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.