DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation

• URL: http://arxiv.org/abs/2606.20663v1
• Date: Wed, 10 Jun 2026 01:16:15 GMT
• Title: DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation
• Abstract summary: We present a pipeline for evaluating AI control protocols to mitigate medication-related harm.<n>Inspired by the medical domain, we argue that safety should account for the severity of unsafe outputs.
• Score: 1.2362187555287152
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models have the potential to expand and improve the access to clinical information by enabling new ways of interacting with medical knowledge in natural language. However, their deployment in medical question-answering settings is safety-critical, since misaligned outputs can lead to severe patient harm. AI control is an emerging approach that introduces external safeguards to mitigate unsafe behaviours in misaligned systems and has been shown to be effective in domains such as code generation. However, its applicability and effectiveness in medical settings have not been systematically studied. In this work, we present a pipeline for evaluating AI control protocols to mitigate medication-related harm. To this end, we introduce DrugBench, an AI control evaluation benchmark which combines 3,671 multi-turn medical conversations from HealthBench with drug information from official FDA labels, covering four categories of medication-related harm: drug interactions, contraindications, dosing constraints, and patient action restrictions. Furthermore, inspired by the medical domain, we argue that safety should account for the severity of unsafe outputs, not just their probability. Under this revised definition, we show that existing control protocols can be subverted and propose severity-based monitoring to address this limitation.

Related papers

• Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops [0.07249400282852114]
  Large Language Models (LLMs) are increasingly deployed in healthcare settings, yet their tendency to hallucinate poses risks when clinical decisions are involved.<n>This study examine whether LLMs recommend recently banned or withdrawn pharmaceuticals when answering clinical questions.<n>We developed a five-agent "Trust but verify" system using a single LLM backbone.
  arXiv  Detail & Related papers  (2026-06-12T06:21:19Z)
• SafeRx-Agent: A Knowledge-Grounded Multi-Agent Framework for Safe and Explainable Medication Recommendation [12.70587762578296]
  We propose SafeRx-Agent, a knowledge-grounded multi-agent framework that uses patient context, external clinical knowledge, and safety verification to recommend traceable medication sets.<n> Experimental results on MIMIC-III and MIMIC-IV datasets show that SafeRx-Agent improves fine-grained medication prediction accuracy while controlling drug interactions, contraindications, and medication set size.
  arXiv  Detail & Related papers  (2026-05-27T22:15:48Z)
• When AI Gets it Wrong: Reliability and Risk in AI-Assisted Medication Decision Systems [0.0]
  In high-risk domains such as medication management, even a single incorrect recommendation can result in severe patient harm.<n>This paper examines the reliability of AI-assisted medication systems by focusing on system failures and their potential clinical consequences.
  arXiv  Detail & Related papers  (2026-04-01T22:53:17Z)
• Medical Malice: A Dataset for Context-Aware Safety in Healthcare LLMs [0.0]
  This work advocates for a shift from universal to context-aware safety.<n>It provides the necessary resources to immunize AI against the nuanced, systemic threats inherent to high-stakes medical environments.
  arXiv  Detail & Related papers  (2025-11-24T11:55:22Z)
• Learning to Describe for Predicting Zero-shot Drug-Drug Interactions [54.172575323610175]
  Adverse drug-drug interactions can compromise the effectiveness of concurrent drug administration. Traditional computational methods for DDI prediction may fail to capture interactions for new drugs due to the lack of knowledge. We propose TextDDI with a language model-based DDI predictor and a reinforcement learning(RL)-based information selector.
  arXiv  Detail & Related papers  (2024-03-13T09:42:46Z)
• ChatFDA: Medical Records Risk Assessment [0.0]
  This study explores a pioneering application aimed at addressing this challenge by assisting caregivers in gauging potential risks derived from medical notes. The application leverages data from openFDA, delivering real-time, actionable insights regarding prescriptions. Preliminary analyses conducted on the MIMIC-III citemimic dataset affirm a proof of concept highlighting a reduction in medical errors and an amplification in patient safety.
  arXiv  Detail & Related papers  (2023-12-20T03:40:45Z)
• Decision Support System for Chronic Diseases Based on Drug-Drug Interactions [67.9840225237587]
  This paper presents a Decision Support System, called DSSDDI, based on drug-drug interactions to support doctors prescribing decisions. The DDI module learns safer and more effective drug representations from the drug-drug interactions. The MD module considers the representations of patients and drugs as context, DDI and patients' similarity as treatment, and medication use as outcome.
  arXiv  Detail & Related papers  (2023-03-04T12:44:38Z)
• Knowledge-Driven New Drug Recommendation [88.35607943144261]
  We develop a drug-dependent multi-phenotype few-shot learner to bridge the gap between existing and new drugs. EDGE eliminates the false-negative supervision signal using an external drug-disease knowledge base. Results show that EDGE achieves 7.3% improvement on the ROC-AUC score over the best baseline.
  arXiv  Detail & Related papers  (2022-10-11T16:07:52Z)
• Principal-Agent Hypothesis Testing [54.154244569974864]
  We consider the relationship between a regulator (the principal) and an experimenter (the agent) such as a pharmaceutical company. The efficacy of the drug is not known to the regulator, so the pharmaceutical company must run a costly trial to prove efficacy to the regulator. We show how to design protocols that are robust to an agent's strategic actions, and derive the optimal protocol in the presence of strategic entrants.
  arXiv  Detail & Related papers  (2022-05-13T17:59:23Z)
• SafeDrug: Dual Molecular Graph Encoders for Safe Drug Recommendations [59.590084937600764]
  We propose a DDI-controllable drug recommendation model named SafeDrug to leverage drugs' molecule structures and model DDIs explicitly. On a benchmark dataset, our SafeDrug is relatively shown to reduce DDI by 19.43% and improves 2.88% on Jaccard similarity between recommended and actually prescribed drug combinations over previous approaches.
  arXiv  Detail & Related papers  (2021-05-05T00:20:48Z)
• COVI White Paper [67.04578448931741]
  Contact tracing is an essential tool to change the course of the Covid-19 pandemic. We present an overview of the rationale, design, ethical considerations and privacy strategy of COVI,' a Covid-19 public peer-to-peer contact tracing and risk awareness mobile application developed in Canada.
  arXiv  Detail & Related papers  (2020-05-18T07:40:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.