PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments

• URL: http://arxiv.org/abs/2605.02240v1
• Date: Mon, 04 May 2026 05:32:25 GMT
• Title: PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
• Abstract summary: Existing medical agent benchmarks primarily focus on static knowledge recall, single-step atomic actions, or action intent without verifiable execution against the environment.<n>PhysicianBench comprises 100 long-horizon tasks adapted from real cases between primary care and physicians.<n>Each task is decomposed into structured checkpoints (670 in total across the benchmark)<n>The best-performing model achieves only 46% success rate (pass@1), while open-source models reach most 19%.
• Score: 5.310457397170786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce PhysicianBench, a benchmark for evaluating LLM agents on physician tasks grounded in real clinical setting within electronic health record (EHR) environments. Existing medical agent benchmarks primarily focus on static knowledge recall, single-step atomic actions, or action intent without verifiable execution against the environment. As a result, they fail to capture the long-horizon, composite workflows that characterize real clinical systems. PhysicianBench comprises 100 long-horizon tasks adapted from real consultation cases between primary care and subspecialty physicians, with each task independently reviewed by a separate panel of physicians. Tasks are instantiated in an EHR environment with real patient records and accessed through the same standard APIs used by commercial EHR vendors. Tasks span 21 specialties (e.g., cardiology, endocrinology, oncology, psychiatry) and diverse workflow types (e.g., diagnosis interpretation, medication prescribing, treatment planning), requiring an average of 27 tool calls per task. Solving each task requires retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. Each task is decomposed into structured checkpoints (670 in total across the benchmark) capturing distinct stages of completion graded by task-specific scripts with execution-grounded verification. Across 13 proprietary and open-source LLM agents, the best-performing model achieves only 46% success rate (pass@1), while open-source models reach at most 19%, revealing a substantial gap between current agent capabilities and the demands of real-world clinical workflows. PhysicianBench provides a realistic and execution-grounded benchmark for measuring progress toward autonomous clinical agents.

Related papers

• The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy [33.20812920007743]
  This work asks what tasks, resistant benchmarks, and interactive training environments are required before medical agents can be trusted in practice.<n>Medical agents are formalized as sequential decision making systems under partial observability.<n>Within this framework, clinical environment scaling, the integration of tools, data, and clinical gyms, is identified as the most actionable yet underexplored direction.
  arXiv  Detail & Related papers  (2026-07-13T07:16:50Z)
• Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases [71.12461204050985]
  MedSP1000 is an SP-derived interactive benchmark for clinical-agent evaluation.<n>It converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubrics.<n>Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios.
  arXiv  Detail & Related papers  (2026-06-03T17:17:16Z)
• EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs [51.129595320595094]
  Clinical decision-making (CDM) is central to real-world clinical, where clinicians infer diagnoses, select treatments, or anticipate future health outcomes under incomplete evidence.<n>LLM models are increasingly used to support these decisions due to strong language capabilities, broad biomedical knowledge, and efficiency.<n>The reliability of LLMs on real-world clinical decision tasks remains insufficiently understood.
  arXiv  Detail & Related papers  (2026-05-28T22:38:26Z)
• A Randomized Controlled Trial and Pilot of Scout: an LLM-Based EHR Search and Synthesis Platform [1.3680778971360474]
  We developed Scout, an EHR search and synthesis platform that enables clinicians to query EHR data using natural language.<n>Scout reduced task completion time by 37.6% and significantly decreased perceived workload.<n>Scout maintained accuracy, completeness, and relevance relative to tasks completed with the EHR-only.
  arXiv  Detail & Related papers  (2026-03-07T22:28:53Z)
• LiveClin: A Live Clinical Benchmark without Leakage [50.45415584327275]
  LiveClin is a live benchmark designed for approximating real-world clinical practice.<n>We transform authentic patient cases into complex, multimodal evaluation scenarios that span the entire clinical pathway.<n>Our evaluation of 26 models on LiveClin reveals the profound difficulty of these real-world scenarios, with the top-performing model achieving a Case Accuracy of just 35.7%.
  arXiv  Detail & Related papers  (2026-02-18T03:59:46Z)
• Timely Clinical Diagnosis through Active Test Selection [49.091903570068155]
  We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design) to better emulate real-world diagnostic reasoning.<n>LLMs act as flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data.<n>We evaluate ACTMED on real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretability, and resource use.
  arXiv  Detail & Related papers  (2025-10-21T18:10:45Z)
• Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks [72.89088985703748]
  The rise of large language models (LLMs) has transformed healthcare by offering clinical guidance, yet their direct deployment to patients poses safety risks.<n>We propose repositioning LLMs as clinical assistants that collaborate with experienced physicians rather than interacting with patients directly.<n>We construct DoctorFLAN, a large-scale Chinese medical dataset comprising 92,000 Q&A instances across 22 clinical tasks and 27 specialties.
  arXiv  Detail & Related papers  (2025-10-13T06:18:27Z)
• Assessing Large Language Models for Structured Medical Order Extraction [0.6015898117103068]
  The MEDIQA-OE 2025 shared task focuses on extracting structured medical orders from extended conversational transcripts.<n>We present the Mason submission, which ranked 5th among 17 participating teams with 105 total submissions.<n>This few-shot configuration achieved an average F1 score of 37.76, with notable improvements in reason and provenance accuracy.
  arXiv  Detail & Related papers  (2025-10-12T06:56:10Z)
• EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysis [62.00431604976949]
  EndoBench is the first comprehensive benchmark specifically designed to assess MLLMs across the full spectrum of endoscopic practice.<n>We benchmark 23 state-of-the-art models, including general-purpose, medical-specialized, and proprietary MLLMs.<n>Our experiments reveal: proprietary MLLMs outperform open-source and medical-specialized models overall, but still trail human experts.
  arXiv  Detail & Related papers  (2025-05-29T16:14:34Z)
• ER-REASON: A Benchmark Dataset for LLM-Based Clinical Reasoning in the Emergency Room [6.910389029249664]
  Large language models (LLMs) have been extensively evaluated on medical question answering tasks based on licensing exams.<n> ER-Reason is a benchmark designed to evaluate LLM-based clinical reasoning and decision-making in the emergency room.
  arXiv  Detail & Related papers  (2025-05-28T22:43:44Z)
• MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks [27.717720332927296]
  We introduce MedAgentBoard, a comprehensive benchmark for the systematic evaluation of multi-agent collaboration, single-LLM, and conventional approaches.<n> MedAgentBoard encompasses four diverse medical task categories: medical (visual) question answering, lay summary generation, structured Electronic Health Record (EHR) predictive modeling, and clinical workflow automation.<n>Our extensive experiments reveal a nuanced landscape: while multi-agent collaboration demonstrates benefits in specific scenarios, it does not consistently outperform advanced single LLMs.
  arXiv  Detail & Related papers  (2025-05-18T11:28:17Z)
• Medchain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence [68.05876437208505]
  We present MedChain, a dataset of 12,163 clinical cases that covers five key stages of clinical workflow.<n>We also propose MedChain-Agent, an AI system that integrates a feedback mechanism and a MCase-RAG module to learn from previous cases and adapt its responses.
  arXiv  Detail & Related papers  (2024-12-02T15:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.