PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

• URL: http://arxiv.org/abs/2607.25485v1
• Date: Tue, 28 Jul 2026 09:24:04 GMT
• Title: PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
• Abstract summary: PatientAgentBench benchmarks patient-facing agentic healthcare.<n>It evaluates a foundation model wrapped in an agent with a sandbox of healthcare tools, conversing with a simulated patient.<n>We benchmarked 10 models across four families on the same 1,200 scenarios and found clinical gaps.
• Score: 5.247635961504091
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Health AI is evolving from answering questions to agentic systems that converse with patients, reason about health records, and act on their behalf. Primary care guards against diagnostic errors and unsafe care; agents assisting in this domain warrant evaluation against the same risks. Current benchmarks focus on medical knowledge, assessed through isolated question-answering or clinician-facing tasks. PatientAgentBench benchmarks patient-facing agentic healthcare; it evaluates a foundation model, wrapped in an agent with a sandbox of healthcare tools, conversing with a simulated patient. Each conversation is scored by an LLM-as-a-Jury across six dimensions via over a hundred conversation-agnostic, clinician-grounded criteria. To validate alignment, licensed clinicians annotated shared conversations, yielding 79-93% adjacent agreement between jury and expert raters, on par with or exceeding clinician inter-rater agreement. We benchmarked 10 models across four families on the same 1,200 scenarios and found clinical gaps. Triage quality is the most discriminating dimension: pass rates rise from 32% for the weakest models to 88% for the strongest, with agents often acting on administrative requests without clinical screening. Clinical safety and workflow accuracy follow the same pattern: the weakest models fail often, fabricating unexecuted actions, while frontier models fail on only 1-3% of cases, from unverified tool outputs and omitted crisis resources in an emergency. More capable models narrow these gaps but do not close them; the strongest scores only 4.25 of 5 overall. These failures surface only in sustained, tool-using conversations against realistic patient records, confirming that static benchmarks are insufficient as healthcare agentic systems gain autonomy. We release the framework as a reproducible, clinician-validated evaluation standard to help the field close this gap.

Related papers

• ResidencyRL: Reinforcement Learning in Simulated Clinical Environments [53.76751756107077]
  We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters.<n>On heldout evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions.<n>Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons.
  arXiv  Detail & Related papers  (2026-08-07T17:04:41Z)
• AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows [7.354281305130301]
  Large language models (LLMs) are increasingly considered for use in clinical consultation tasks.<n>We propose AIPatient Arena, an EHRs-grounded evaluation framework for assessing the clinical utility of LLMs.
  arXiv  Detail & Related papers  (2026-06-16T03:35:17Z)
• AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment [3.1849948640573245]
  AcuityBench is a benchmark for evaluating whether language models identify the appropriate urgency of care from user medical presentations.<n>It consists of five public datasets spanning user conversations, online forum posts, clinical vignettes, and patient portal messages.<n>It supports two complementary task formats: explicit four-way classification in a QA setting, and free-form conversational responses evaluated with a rubric-based judge.
  arXiv  Detail & Related papers  (2026-05-12T01:39:46Z)
• VeriSim: A Configurable Framework for Evaluating Medical AI Under Realistic Patient Noise [2.997608017642323]
  We introduce VeriSim, a truth-preserving patient simulation framework.<n>Our framework injects controllable, clinically evidence-grounded noise into patient responses.<n>Our results highlight a critical Sim-to-Real gap in current medical AI.
  arXiv  Detail & Related papers  (2026-04-12T03:35:22Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• From Days to Minutes: An Autonomous AI Agent Achieves Reliable Clinical Triage in Remote Patient Monitoring [2.0918370570198763]
  Remote patient monitoring ( RPM) generates vast data, yet landmark trials (Tele-HF, BEAT-HF) failed because data volume overwhelmed clinical staff.<n>We developed Sentinel, an autonomous AI agent using Model Context Protocol (MCP) for contextual triage of RPM vitals.
  arXiv  Detail & Related papers  (2026-03-10T00:50:54Z)
• ART: Action-based Reasoning Task Benchmarking for Medical AI Agents [0.0]
  We introduce Action-based Reasoning clinical Task benchmark for medical AI agents.<n>We identify three dominant error categories: retrieval failures, aggregation errors, and conditional logic misjudgments.<n>Our four-stage pipeline produces diverse, clinically validated tasks grounded in real patient data.
  arXiv  Detail & Related papers  (2026-01-13T21:26:11Z)
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services [49.70819009392778]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) offer opportunities to augment dispatchers.<n>This study aimed to develop and evaluate a taxonomy-grounded, multi-agent system for simulating realistic scenarios.
  arXiv  Detail & Related papers  (2025-10-24T08:01:21Z)
• Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models [57.73472878679636]
  We introduce Med-RewardBench, the first benchmark specifically designed to evaluate medical reward models and judges.<n>Med-RewardBench features a multimodal dataset spanning 13 organ systems and 8 clinical departments, with 1,026 expert-annotated cases.<n>A rigorous three-step process ensures high-quality evaluation data across six clinically critical dimensions.
  arXiv  Detail & Related papers  (2025-08-29T08:58:39Z)
• Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language Models [87.66870367661342]
  Large language models (LLMs) are used in AI applications in healthcare.<n>Red-teaming framework that continuously stress-test LLMs can reveal significant weaknesses in four safety-critical domains.<n>A suite of adversarial agents is applied to autonomously mutate test cases, identify/evolve unsafe-triggering strategies, and evaluate responses.<n>Our framework delivers an evolvable, scalable, and reliable safeguard for the next generation of medical AI.
  arXiv  Detail & Related papers  (2025-07-30T08:44:22Z)
• AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments [2.567146936147657]
  We introduce AgentClinic, a multimodal agent benchmark for evaluating large language models (LLM) in simulated clinical environments.<n>We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy.
  arXiv  Detail & Related papers  (2024-05-13T17:38:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.