AI Safety Training Can be Clinically Harmful

• URL: http://arxiv.org/abs/2604.23445v1
• Date: Sat, 25 Apr 2026 21:09:50 GMT
• Title: AI Safety Training Can be Clinically Harmful
• Abstract summary: We evaluate four generative models on 250 Prolonged Exposure therapy scenarios and 146 CBT cognitive restructuring exercises.<n>All models scored near-perfectly on surface acknowledgment (0.91-1.00) while therapeutic appropriateness collapsed to 0.22-0.33 at the highest severity for three of four models.
• Score: 15.31927635809423
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are being deployed as mental health support agents at scale, yet only 16% of LLM-based chatbot interventions have undergone rigorous clinical efficacy testing, and simulations reveal psychological deterioration in over one-third of cases. We evaluate four generative models on 250 Prolonged Exposure (PE) therapy scenarios and 146 CBT cognitive restructuring exercises (plus 29 severity-escalated variants), scored by a three-judge LLM panel. All models scored near-perfectly on surface acknowledgment (~0.91-1.00) while therapeutic appropriateness collapsed to 0.22-0.33 at the highest severity for three of four models, with protocol fidelity reaching zero for two. Under CBT severity escalation, one model's task completeness dropped from 92% to 71% while the frontier model's safety-interference score fell from 0.99 to 0.61. We identify a systematic, modality-spanning failure: RLHF safety alignment disrupts the therapeutic mechanism of action by grounding patients during imaginal exposure, offering false reassurance, inserting crisis resources into controlled exercises, and refusing to challenge distorted cognitions mentioning self-harm in PE; and through task abandonment or safety-preamble insertion during CBT cognitive restructuring. These findings motivate a five-axis evaluation framework (protocol fidelity, hallucination risk, behavioral consistency, crisis safety, demographic robustness), mapped onto FDA SaMD and EU AI Act requirements. We argue that no AI mental health system should proceed to deployment without passing multi-axis evaluation across all five dimensions.

Related papers

• IyawoBench v2.0: Extended Diagnostic Evaluation of Large Language Model Clinical Triage in Nigerian Primary Care [0.0]
  We present IyawoBench v2.0, an extended diagnostic evaluation of large language model clinical triage.<n>We introduce a formal mathematical framework that decomposes triage safety into three distinct failure modes: Conservative Escalation Bias, Systematic Downgrade Bias, and Middle-Tier Instability.
  arXiv  Detail & Related papers  (2026-07-31T07:11:55Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• Predicting Neuromodulation Outcome for Parkinson's Disease with Generative Virtual Brain Model [95.41752463487008]
  Parkinson's disease affects over ten million people worldwide.<n>Inter-individual variability limits empirical treatment selection, increasing non-negligible surgical risk and cost.<n>We bridge this gap with a pretraining-finetuning framework to predict outcomes directly from resting-state fMRI.
  arXiv  Detail & Related papers  (2026-03-31T02:36:10Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models [54.08784776767683]
  We present Physical-Conditioned World Model Attack (PhysCond-WMA), the first white-box world model attack that perturbs physical-condition channels.<n>PhysCond-WMA induces semantic, logic, or decision-level distortion while preserving perceptual fidelity.
  arXiv  Detail & Related papers  (2026-02-21T07:22:37Z)
• Advancing AI Trustworthiness Through Patient Simulation: Risk Assessment of Conversational Agents for Antidepressant Selection [1.90974530523188]
  Patient simulator generates realistic, controllable patient interactions that vary across medical, linguistic, and behavioral dimensions.<n>Simulator allows annotators and an independent AI judge to assess agent performance, identify hallucinations and inaccuracies, and characterize risk patterns across diverse patient populations.
  arXiv  Detail & Related papers  (2026-02-11T21:53:18Z)
• SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care [2.2245087144264657]
  Large language models (LLMs) show promise in clinical decision support yet risk acquiescing to patient pressure for inappropriate care.<n>We introduce SycoEval-EM, a multi-agent simulation framework evaluating robustness through adversarial patient persuasion.
  arXiv  Detail & Related papers  (2026-01-23T08:01:39Z)
• HARBOR: Holistic Adaptive Risk assessment model for BehaviORal healthcare [4.268975608471992]
  We introduce HARBOR, a behavioral health aware language model designed to predict a discrete mood and risk score.<n>We also release PEARL, a longitudinal behavioral healthcare dataset spanning four years of monthly observations from three patients.
  arXiv  Detail & Related papers  (2025-12-21T17:27:10Z)
• Breast Cancer Recurrence Risk Prediction Based on Multiple Instance Learning [0.0]
  This study investigates the potential of computational pathology to stratify patients using deep learning on routine Hematoxylin and Eosin stained whole-slide images (WSIs)<n>We developed and compared three Multiple Instance Learning frameworks -- CLAM-SB, ABMIL, and ConvNeXt-MIL-XGBoost -- on an in-house dataset of 210 patient cases.<n>In a 5-fold cross-validation, the modified CLAM-SB model demonstrated the strongest performance, achieving a mean Area Under the Curve (AUC) of 0.836 and a classification accuracy of 76.2%.
  arXiv  Detail & Related papers  (2025-12-21T13:46:50Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language Models [87.66870367661342]
  Large language models (LLMs) are used in AI applications in healthcare.<n>Red-teaming framework that continuously stress-test LLMs can reveal significant weaknesses in four safety-critical domains.<n>A suite of adversarial agents is applied to autonomously mutate test cases, identify/evolve unsafe-triggering strategies, and evaluate responses.<n>Our framework delivers an evolvable, scalable, and reliable safeguard for the next generation of medical AI.
  arXiv  Detail & Related papers  (2025-07-30T08:44:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.