From Black-Box Confidence to Measurable Trust in Clinical AI: A Framework for Evidence, Supervision, and Staged Autonomy

• URL: http://arxiv.org/abs/2604.26671v1
• Date: Wed, 29 Apr 2026 13:40:14 GMT
• Title: From Black-Box Confidence to Measurable Trust in Clinical AI: A Framework for Evidence, Supervision, and Staged Autonomy
• Abstract summary: Trust in clinical artificial intelligence (AI) cannot be reduced to model accuracy, fluency of generation, or overall positive user impression.<n>This article proposes a framework for trustworthy clinical AI built around three principles: evidence, supervision, and staged autonomy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Trust in clinical artificial intelligence (AI) cannot be reduced to model accuracy, fluency of generation, or overall positive user impression. In medicine, trust must be engineered as a measurable system property grounded in evidence, supervision, and operational boundaries of AI autonomy. This article proposes a practical framework for trustworthy clinical AI built around three principles: evidence, supervision, and staged autonomy. Rather than replacing deterministic clinical logic wholesale with end-to-end black-box models, the proposed approach combines a deterministic core, a patient-specific AI assistant for contextual validation, a multi-tier model escalation mechanism, and a human supervision layer for verification, escalation, and risk control. We demonstrate that trust also depends on selective verification of clinically critical findings, bounded clinical context, disciplined prompt architecture, and careful evaluation on realistic cases. Classifier-driven modular prompting is examined as an incremental path to scaling clinical depth without sacrificing prompt performance and without waiting for complete rule-based coverage. To operationalize trust, a set of trust metrics is proposed, built on metrological principles -- measurement uncertainty, calibration, traceability -- enabling quantitative rather than subjective assessment of each architectural layer. In this perspective, trustworthy clinical AI emerges not as a property of an individual model, but as an architectural outcome of a system into which evidence trails, human oversight, tiered escalation, and graduated action rights are embedded from the outset.

Related papers

• AI Morbidity and Mortality: A Framework for Clinical AI Failure Review [4.342185705620882]
  We propose AI Morbidity and Mortality (AI M&M), a structured, blameless framework for case-based review of clinical AI failures.<n>The framework combines standardized case intake, evidence preservation and investigator-level reconstruction, tool-in-loop attribution, and corrective-action tracking.<n>It is intended to complement, rather than replace, model monitoring, patient safety reporting, and regulatory oversight by converting individual AI-in-workflow failures into actionable institutional learning.
  arXiv  Detail & Related papers  (2026-08-31T07:41:30Z)
• Algorithmic Authority and the Clinical Standard of Care [5.584060970507507]
  I argue that the architecture of clinical AI systems already functions as de facto medical regulation.<n>I propose a dialectical standard of care that treats the integrated AI-physician dyad as the singular responsible diagnostic entity.
  arXiv  Detail & Related papers  (2026-04-29T00:36:20Z)
• Towards Trustworthy Depression Estimation via Disentangled Evidential Learning [50.22167852149165]
  We propose EviDep, an evidential learning framework that jointly quantifies depression severity.<n>EviDep enforces strict information integrity to guarantee robust evidence synthesis.<n>It achieves state-of-the-art predictive accuracy and superior uncertainty calibration, delivering a robust fail-safe mechanism for trustworthy clinical screening.
  arXiv  Detail & Related papers  (2026-04-17T13:27:11Z)
• Grounding Clinical AI Competency in Human Cognition Through the Clinical World Model and Skill-Mix Framework [2.048286515359845]
  We introduce the Clinical World Model, a framework that formalizes care as a tripartite interaction.<n>We develop parallel decision-making architectures for providers, patients, and AI agents.<n>The framework supplies a common grammar through which clinical AI can be specified, evaluated, and bounded across stakeholders.
  arXiv  Detail & Related papers  (2026-04-09T13:20:13Z)
• Agentic Cognitive Profiling: Realigning Automated Alzheimer's Disease Detection with Clinical Construct Validity [66.94391219005291]
  This paper proposes Agentic Cognitive Profiling (ACP), an agentic framework that realigns automated screening with clinical protocol logic.<n>Central to our design is decoupling semantic understanding from measurement by delegating all quantification to deterministic function calling.<n>ACP achieves 90.5% score match rate in task examination and 85.3% accuracy in AD prediction, surpassing popular baselines.
  arXiv  Detail & Related papers  (2026-03-18T06:15:35Z)
• Overalignment in Frontier LLMs: An Empirical Study of Sycophantic Behaviour in Healthcare [1.9010852820067994]
  We propose the Adjusted Sycophancy Score, a novel metric that isolates alignment bias by accounting for model instability, or "confusability"<n>Our results suggest that benchmark performance is not a proxy for clinical reliability, and that simplified reasoning structures may offer superior robustness against expert-driven sycophancy.
  arXiv  Detail & Related papers  (2026-01-26T10:21:34Z)
• "Crash Test Dummies" for AI-Enabled Clinical Assessment: Validating Virtual Patient Scenarios with Virtual Learners [0.0]
  In medical and health professions education, AI is increasingly used to assess clinical competencies, including via virtual standardized patients.<n>Most evaluations rely on AI-human interrater reliability and lack a measurement framework for how cases, learners, and raters jointly shape scores.<n>We develop an open-source platform and measurement model for robust competency evaluation across cases and rating conditions.
  arXiv  Detail & Related papers  (2026-01-26T02:47:28Z)
• AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning [73.50200033931148]
  We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists.<n>By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback.<n> Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations.
  arXiv  Detail & Related papers  (2026-01-23T11:59:13Z)
• Bridging the Knowledge-Action Gap by Evaluating LLMs in Dynamic Dental Clinical Scenarios [9.865786198063644]
  The transition of Large Language Models (LLMs) from passive knowledge retrievers to autonomous clinical agents demands a shift in evaluation-from static accuracy to dynamic behavioral reliability.<n>This study empirically charts the capability boundaries of dental LLMs, providing a roadmap for bridging the gap between standardized knowledge and safe, autonomous clinical practice.
  arXiv  Detail & Related papers  (2026-01-19T11:36:39Z)
• Interpretable Clinical Classification with Kolgomorov-Arnold Networks [70.72819760172744]
  Kolmogorov-Arnold Networks (KANs) offer intrinsic interpretability through transparent, symbolic representations.<n>KANs support built-in patient-level insights, intuitive visualizations, and nearest-patient retrieval.<n>These results position KANs as a promising step toward trustworthy AI that clinicians can understand, audit, and act upon.
  arXiv  Detail & Related papers  (2025-09-20T17:21:58Z)
• Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models [52.2001050216955]
  Existing methods aim to enhance the performance of Medical Vision Language Model (MedVLM) by adjusting model structure, fine-tuning with high-quality data, or through preference fine-tuning.<n>We propose an expert-in-the-loop framework named Expert-Controlled-Free Guidance (Expert-CFG) to align MedVLM with clinical expertise without additional training.
  arXiv  Detail & Related papers  (2025-07-12T09:03:30Z)
• The challenge of uncertainty quantification of large language models in medicine [0.0]
  This study investigates uncertainty quantification in large language models (LLMs) for medical applications.<n>Our research frames uncertainty not as a barrier but as an essential part of knowledge that invites a dynamic and reflective approach to AI design.
  arXiv  Detail & Related papers  (2025-04-07T17:24:11Z)
• TREEMENT: Interpretable Patient-Trial Matching via Personalized Dynamic Tree-Based Memory Network [54.332862955411656]
  Clinical trials are critical for drug development but often suffer from expensive and inefficient patient recruitment. In recent years, machine learning models have been proposed for speeding up patient recruitment via automatically matching patients with clinical trials. We introduce a dynamic tree-based memory network model named TREEMENT to provide accurate and interpretable patient trial matching.
  arXiv  Detail & Related papers  (2023-07-19T12:35:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.