Deployment and Evaluation of an EHR-integrated, Large Language Model-Powered Tool to Triage Surgical Patients

• URL: http://arxiv.org/abs/2603.17234v1
• Date: Wed, 18 Mar 2026 00:36:58 GMT
• Title: Deployment and Evaluation of an EHR-integrated, Large Language Model-Powered Tool to Triage Surgical Patients
• Abstract summary: Surgical co-management (SCM) is an evidence-based model in which hospitalists jointly manage patients alongside surgical teams.<n>SCM Navigator categorized patients as appropriate, not appropriate, or possibly appropriate for SCM.<n>Since deployment, 6,193 cases have been triaged, of which 1,582 (23%) were recommended for hospitalist consultation.
• Score: 3.381441580878978
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Surgical co-management (SCM) is an evidence-based model in which hospitalists jointly manage medically complex perioperative patients alongside surgical teams. Despite its clinical and financial value, SCM is limited by the need to manually identify eligible patients. To determine whether SCM triage can be automated, we conducted a prospective, unblinded study at Stanford Health Care in which an LLM-based, electronic health record (EHR)-integrated triage tool (SCM Navigator) provided SCM recommendations followed by physician review. Using pre-operative documentation, structured data, and clinical criteria for perioperative morbidity, SCM Navigator categorized patients as appropriate, not appropriate, or possibly appropriate for SCM. Faculty indicated their clinical judgment and provided free-text feedback when they disagreed. Sensitivity, specificity, positive predictive value, and negative predictive value were measured using physician determinations as a reference. Free-text reasons were thematically categorized, and manual chart review was conducted on all false-negative cases and 30 randomly selected cases from the largest false-positive category. Since deployment, 6,193 cases have been triaged, of which 1,582 (23%) were recommended for hospitalist consultation. SCM Navigator displayed high sensitivity (0.94, 95% CI 0.91-0.96) and moderate specificity (0.74, 95% CI 0.71-0.77). Post-hoc chart review suggested most discrepancies reflect modifiable gaps in clinical criteria, institutional workflow, or physician practice variability rather than LLM misclassification, which accounted for 2 of 19 (11%) false-negative cases. These findings demonstrate that an LLM-powered, EHR-integrated, human-in-the-loop AI system can accurately and safely triage surgical patients for SCM, and that AI-enabled screening tools can augment and potentially automate time-intensive clinical workflows.

Related papers

• Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking [27.829988205900055]
  We introduce MedQADE, the first standardised open-response clinical benchmark for German.<n>The top-performing evaluator model, Gemini 3 Flash, reached alignment consistent with the physician ceiling.<n>Despite this statistical alignment, automated evaluators exhibited near-absent clinical metacognition.
  arXiv  Detail & Related papers  (2026-07-01T15:55:31Z)
• Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases [71.12461204050985]
  MedSP1000 is an SP-derived interactive benchmark for clinical-agent evaluation.<n>It converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubrics.<n>Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios.
  arXiv  Detail & Related papers  (2026-06-03T17:17:16Z)
• Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters [3.018184429993625]
  Methods requiring expert review per scoring instance are too slow and expensive for safe, iterative deployment.<n>Twenty clinicians authored 1,646 rubrics for 823 clinical cases across primary care, psychiatry, oncology, and behavioral health.<n>Case-specific rubrics offer a path for clinical AI evaluation that preserves expert judgment while enabling automation at three orders lower cost.
  arXiv  Detail & Related papers  (2026-04-27T17:17:56Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences [50.71326426975699]
  We present MedArena, an interactive evaluation platform for medical large language models (LLMs)<n>Given a clinician-provided query, MedArena presents responses from two randomly selected models and asks the user to select the preferred response.<n>Out of 1571 preferences collected across 12 LLMs up to November 1, 2025, Gemini 2.0 Flash Thinking, Gemini 2.5 Pro, and GPT-4o were the top three models by Bradley-Terry rating.
  arXiv  Detail & Related papers  (2026-03-13T22:30:26Z)
• Machine Learning Based Prediction of Surgical Outcomes in Chronic Rhinosinusitis from Clinical Data [1.5122833700794007]
  Chronic rhinosinusitis (CRS) imposes a substantial burden on quality of life (QoL) and societal cost.<n>In this study, we evaluated supervised machine learning models for predicting surgical benefit in CRS.
  arXiv  Detail & Related papers  (2026-02-19T22:47:50Z)
• Patient-Centered Summarization Framework for AI Clinical Summarization: A Mixed-Methods Design [23.21070690395588]
  We propose a new standard for Artificial Intelligence (AI) clinical summarization tasks: Patient-Centered Summaries (PCS)<n>Our objective was to develop a framework to generate PCS that capture patient values and ensure clinical utility.<n>Five open-source LLMs generated summaries for 72 consultations using zero-shot and few-shot prompting.
  arXiv  Detail & Related papers  (2025-10-31T15:08:18Z)
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services [49.70819009392778]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) offer opportunities to augment dispatchers.<n>This study aimed to develop and evaluate a taxonomy-grounded, multi-agent system for simulating realistic scenarios.
  arXiv  Detail & Related papers  (2025-10-24T08:01:21Z)
• LGE-Guided Cross-Modality Contrastive Learning for Gadolinium-Free Cardiomyopathy Screening in Cine CMR [51.11296719862485]
  We propose a Contrastive Learning and Cross-Modal alignment framework for gadolinium-free cardiomyopathy screening using cine CMR sequences.<n>By aligning the latent spaces of cine CMR and Late Gadolinium Enhancement (LGE) sequences, our model encodes fibrosis-specific pathology into cine CMR embeddings.
  arXiv  Detail & Related papers  (2025-08-23T07:21:23Z)
• MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks [47.486705282473984]
  Large language models (LLMs) achieve near-perfect scores on medical exams.<n>These evaluations inadequately reflect complexity and diversity of real-world clinical practice.<n>We introduce MedHELM, an evaluation framework for assessing LLM performance for medical tasks.
  arXiv  Detail & Related papers  (2025-05-26T22:55:49Z)
• Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases [48.87360916431396]
  We introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references.<n>We propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey.<n>Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc.
  arXiv  Detail & Related papers  (2025-03-06T18:35:39Z)
• Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm [15.627870862369784]
  Large language models (LLMs) are gaining increasing interests to improve clinical efficiency for medical diagnosis. We propose an automatic evaluation paradigm tailored to assess the LLMs' capabilities in delivering clinical services.
  arXiv  Detail & Related papers  (2024-03-25T06:17:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.