Safety and accuracy follow different scaling laws in clinical large language models

• URL: http://arxiv.org/abs/2605.04039v1
• Date: Tue, 05 May 2026 17:57:19 GMT
• Title: Safety and accuracy follow different scaling laws in clinical large language models
• Abstract summary: RadSaFE-200 is a benchmark of 200 multiple-choice questions with clinician-defined clean evidence, conflict evidence, and option-level labels for high-risk error, unsafe answer, and evidence contradiction.<n>Clean evidence produced the strongest improvement, increasing mean accuracy from 73.5% to 94.1%, while reducing high-risk error from 12.0% to 2.6%.<n>Standard RAG and agentic RAG did not reproduce this safety profile, but high-risk error and dangerous overconfidence remained elevated.
• Score: 4.212844425331427
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinical LLMs are often scaled by increasing model size, context length, retrieval complexity, or inference-time compute, with the implicit expectation that higher accuracy implies safer behavior. This assumption is incomplete in medicine, where a few confident, high-risk, or evidence-contradicting errors can matter more than average benchmark performance. We introduce SaFE-Scale, a framework for measuring how clinical LLM safety changes across model scale, evidence quality, retrieval strategy, context exposure, and inference-time compute. To instantiate this framework, we introduce RadSaFE-200, a Radiology Safety-Focused Evaluation benchmark of 200 multiple-choice questions with clinician-defined clean evidence, conflict evidence, and option-level labels for high-risk error, unsafe answer, and evidence contradiction. We evaluated 34 locally deployed LLMs across six deployment conditions: closed-book prompting (zero-shot), clean evidence, conflict evidence, standard RAG, agentic RAG, and max-context prompting. Clean evidence produced the strongest improvement, increasing mean accuracy from 73.5% to 94.1%, while reducing high-risk error from 12.0% to 2.6%, contradiction from 12.7% to 2.3%, and dangerous overconfidence from 8.0% to 1.6%. Standard RAG and agentic RAG did not reproduce this safety profile: agentic RAG improved accuracy over standard RAG and reduced contradiction, but high-risk error and dangerous overconfidence remained elevated. Max-context prompting increased latency without closing the safety gap, and additional inference-time compute produced only limited gains. Worst-case analysis showed that clinically consequential errors concentrated in a small subset of questions. Clinical LLM safety is therefore not a passive consequence of scaling, but a deployment property shaped by evidence quality, retrieval design, context construction, and collective failure behavior.

Related papers

• When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information [14.659028787864996]
  Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.<n>We conduct a behavioral analysis of LLMs under clinical information uncertainty.<n>We observe significant variation across models in their ability to abstain when the correct answer is unavailable.
  arXiv  Detail & Related papers  (2026-08-10T03:28:46Z)
• Retrieval-Augmented Generation in LLMs for Mental Health: Quantifying the Incremental Contribution of Retrieval Within a Layered Safety Architecture [1.9213555198905314]
  Digital mental health interventions (DMHIs) offer scalable support, but ensuring they accurately detect users' intent during volatile situations can be challenging.<n>This paper evaluates six Large Language models within a DMHI called Wysa, via a controlled comparison of RAG-enabled versus RAG-disabled modes.<n>The study computed classification accuracy, recall, precision and F1 scores against ground truth labels and tested differences for statistical significance.
  arXiv  Detail & Related papers  (2026-07-17T06:32:12Z)
• A safety-oriented hypothetico-deductive framework for AI-assisted differential diagnosis [22.43455890575628]
  We present AegisDx, a safety-oriented framework for hypothetico-deductive clinical reasoning.<n>AegisDx generates broad differential diagnoses, enforces explicit screening for dangerous "must-not-miss" conditions.<n>AegisDx improved the physician-rated composite safety score from 4.31 to 4.55 on a 5-point scale.
  arXiv  Detail & Related papers  (2026-07-09T01:30:24Z)
• Improving Model Safety by Targeted Error Correction [10.82789277277678]
  We introduce a dual-classifier GBDT pipeline to distinguish human-like errors from high-risk non-human misclassifications.<n>Our results confirm the pipeline introduces negligible inference (1.60% overhead for the animal dataset, 1.84% for ISIC, and 1.70% for SICAPv2)<n>Our conservative correction strategy successfully reduced dangerous nonhuman errors by 34.1% in ISIC and 12.57% in SICAPv2.
  arXiv  Detail & Related papers  (2026-05-04T12:47:41Z)
• A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Models [34.38896321208398]
  Large language models (LLMs) are increasingly deployed across healthcare, yet existing benchmarks fail to capture model behavior under adversarial or ethically complex conditions.<n>We developed a multi-domain red teaming framework evaluating eleven contemporary LLMs across 690 clinically grounded scenarios.<n>Results revealed substantial performance variance, with mean scores ranging from 0.791 to 0.984.
  arXiv  Detail & Related papers  (2026-04-15T14:57:55Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization [50.71047638695205]
  Reasoning language models (RLMs) are increasingly used in programming.<n>Yet, even state-of-the-art RLMs frequently introduce critical security vulnerabilities in generated code.<n>We present SecPI, a fine-tuning pipeline that teaches RLMs to internalize structured security reasoning.
  arXiv  Detail & Related papers  (2026-04-04T04:29:11Z)
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis [96.92417622318267]
  ATBench is a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety.<n>It organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm.<n>It contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools.
  arXiv  Detail & Related papers  (2026-04-02T13:26:20Z)
• Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification [60.18369393468405]
  Existing verifiers usually underperform owing to a lack of domain knowledge and limited calibration.<n>GLEAN compiles expert-curated protocols into trajectory-informed, well-calibrated correctness signals.<n>We empirically validate GLEAN with agentic clinical diagnosis across three diseases from the MIMIC-IV dataset.
  arXiv  Detail & Related papers  (2026-03-03T09:36:43Z)
• URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Models [35.441039437111606]
  URAG is a benchmark designed to assess the uncertainty of RAG systems across various fields like healthcare, programming, science, math, and general text.<n>We apply the evaluation pipeline to 8 standard RAG methods, measuring their performance through both accuracy and prediction-set sizes based on LAC and APS metrics.
  arXiv  Detail & Related papers  (2026-03-02T00:22:06Z)
• A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing [0.4349324020366305]
  Large language models (LLMs) show promise for healthcare question answering, but clinical use is limited by weak verification, insufficient evidence grounding, and unreliable confidence signalling.<n>We propose a multi-agent medical QA framework that combines complementary LLMs with evidence retrieval, uncertainty estimation, and bias checks to improve answer reliability.
  arXiv  Detail & Related papers  (2026-02-15T14:17:27Z)
• Diagnosing Hallucination Risk in AI Surgical Decision-Support: A Sequential Framework for Sequential Validation [5.469454486414467]
  Large language models (LLMs) offer transformative potential for clinical decision support in spine surgery.<n>LLMs pose significant risks through hallucinations, which are factually inconsistent or contextually misaligned outputs.<n>This study introduces a clinician-centered framework to quantify hallucination risks by evaluating diagnostic precision, recommendation quality, reasoning robustness, output coherence, and knowledge alignment.
  arXiv  Detail & Related papers  (2025-11-01T15:25:55Z)
• Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language Models [87.66870367661342]
  Large language models (LLMs) are used in AI applications in healthcare.<n>Red-teaming framework that continuously stress-test LLMs can reveal significant weaknesses in four safety-critical domains.<n>A suite of adversarial agents is applied to autonomously mutate test cases, identify/evolve unsafe-triggering strategies, and evaluate responses.<n>Our framework delivers an evolvable, scalable, and reliable safeguard for the next generation of medical AI.
  arXiv  Detail & Related papers  (2025-07-30T08:44:22Z)
• Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies [11.0505830548286]
  This study introduces a granular error taxonomy through systematic analysis of top 10 models on MedBench.<n> Evaluation of 10 leading models reveals vulnerabilities, despite achieving 0.86 accuracy in medical knowledge recall.<n>Our analysis uncovers systemic weaknesses in knowledge boundary enforcement and multi-step reasoning.
  arXiv  Detail & Related papers  (2025-03-10T13:28:25Z)
• Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework [77.45983464131977]
  We focus on how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications.<n>Our research identifies two critical latent factors affecting RAG's confidence in its predictions.<n>We develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers.
  arXiv  Detail & Related papers  (2024-09-24T14:52:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.