A Randomized Controlled Trial and Pilot of Scout: an LLM-Based EHR Search and Synthesis Platform

• URL: http://arxiv.org/abs/2604.26953v1
• Date: Sat, 07 Mar 2026 22:28:53 GMT
• Title: A Randomized Controlled Trial and Pilot of Scout: an LLM-Based EHR Search and Synthesis Platform
• Abstract summary: We developed Scout, an EHR search and synthesis platform that enables clinicians to query EHR data using natural language.<n>Scout reduced task completion time by 37.6% and significantly decreased perceived workload.<n>Scout maintained accuracy, completeness, and relevance relative to tasks completed with the EHR-only.
• Score: 1.3680778971360474
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Clinical documentation and data retrieval within Electronic Health Records (EHRs) contribute substantially to clinician workload and burnout. To address this, we developed Scout, an LLM-based EHR search and synthesis platform that enables clinicians to query EHR data using natural language. Each response includes citations linking each claim to the original data source, facilitating easy verification of generated content. We conducted a prospective randomized, evaluator-blinded crossover trial across seven clinical specialties (20 participants, 200 structured cases). Participants completed realistic clinical tasks using either Scout or the EHR alone, with outcomes including time to completion, NASA Task Load Index workload scores, and blinded expert adjudication of accuracy, completeness, and relevance. Scout reduced task completion time by 37.6% and significantly decreased perceived workload, with the largest reductions in mental demand, effort, and temporal demand. Non-inferiority analyses showed that tasks completed with Scout maintained accuracy, completeness, and relevance relative to tasks completed with the EHR-only. A concurrent pilot deployment across over 200 users and more than 20 specialties generated over 6,600 interactions in three months, revealing diverse clinical and administrative use cases. Automated evaluation using an LLM-as-judge framework identified errors at low rates. Subsequent manual review of a subset of outputs revealed that most claims flagged by the automated judge as errors were in fact supported by the patient chart, demonstrating the importance of human validation. These findings provide early trial-based evidence that LLM-powered EHR tools can meaningfully reduce clinical and administrative workloads while maintaining output quality.

Related papers

• EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs [51.129595320595094]
  Clinical decision-making (CDM) is central to real-world clinical, where clinicians infer diagnoses, select treatments, or anticipate future health outcomes under incomplete evidence.<n>LLM models are increasingly used to support these decisions due to strong language capabilities, broad biomedical knowledge, and efficiency.<n>The reliability of LLMs on real-world clinical decision tasks remains insufficiently understood.
  arXiv  Detail & Related papers  (2026-05-28T22:38:26Z)
• PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments [5.310457397170786]
  Existing medical agent benchmarks primarily focus on static knowledge recall, single-step atomic actions, or action intent without verifiable execution against the environment.<n>PhysicianBench comprises 100 long-horizon tasks adapted from real cases between primary care and physicians.<n>Each task is decomposed into structured checkpoints (670 in total across the benchmark)<n>The best-performing model achieves only 46% success rate (pass@1), while open-source models reach most 19%.
  arXiv  Detail & Related papers  (2026-05-04T05:32:25Z)
• Detecting Clinical Discrepancies in Health Coaching Agents: A Dual-Stream Memory and Reconciliation Architecture [71.46525715889656]
  General-purpose agent memory systems optimize for coherence by overwriting older facts with the user's latest statement.<n>We introduce a Dual-Stream Memory Architecture that strictly separates the patient narrative from the structured clinical record.<n>We evaluate this architecture on 26 patients across 675 longitudinal wellness coaching sessions.
  arXiv  Detail & Related papers  (2026-04-29T17:59:28Z)
• Agentic Cognitive Profiling: Realigning Automated Alzheimer's Disease Detection with Clinical Construct Validity [66.94391219005291]
  This paper proposes Agentic Cognitive Profiling (ACP), an agentic framework that realigns automated screening with clinical protocol logic.<n>Central to our design is decoupling semantic understanding from measurement by delegating all quantification to deterministic function calling.<n>ACP achieves 90.5% score match rate in task examination and 85.3% accuracy in AD prediction, surpassing popular baselines.
  arXiv  Detail & Related papers  (2026-03-18T06:15:35Z)
• EHRNavigator: A Multi-Agent System for Patient-Level Clinical Question Answering over Heterogeneous Electronic Health Records [31.559633376006442]
  EHRNavigator is a multi-agent framework that harnesses AI agents to perform patient-level question answering across heterogeneous and multimodal EHR data.<n>It achieved 86% accuracy on real-world cases while maintaining clinically acceptable response times.
  arXiv  Detail & Related papers  (2026-01-15T03:02:15Z)
• Real-world validation of a multimodal LLM-powered pipeline for High-Accuracy Clinical Trial Patient Matching leveraging EHR data [0.0]
  Patient recruitment in clinical trials is hindered by complex eligibility criteria and labor-intensive chart reviews.<n>We introduce an integration-free, LLM-powered pipeline that automates patient-trial matching using unprocessed documents extracted from EHRs.<n>Our approach leverages (1) the new reasoning-LLM paradigm, enabling the assessment of even the most complex criteria, (2) visual capabilities of latest LLMs to interpret medical records without lossy image-to-text conversions, and (3) multimodal embeddings for efficient medical record search.
  arXiv  Detail & Related papers  (2025-03-19T16:12:11Z)
• Almanac Copilot: Towards Autonomous Electronic Health Record Navigation [4.939203290543146]
  Clinicians spend large amounts of time on clinical documentation. Transition from paper-based records has been negatively associated with clinician wellness. We present Almanac Copilot, an autonomous agent capable of assisting clinicians with EMR-specific tasks.
  arXiv  Detail & Related papers  (2024-04-30T22:55:27Z)
• EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries [9.031182965159976]
  Large Language Models (LLMs) show promise in efficiently analyzing vast and complex data. We introduce EHRNoteQA, a novel benchmark built on the MIMIC-IV EHR, comprising 962 different QA pairs each linked to distinct patients' discharge summaries. EHRNoteQA includes questions that require information across multiple discharge summaries and covers eight diverse topics, mirroring the complexity and diversity of real clinical inquiries.
  arXiv  Detail & Related papers  (2024-02-25T09:41:50Z)
• MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records [60.35217378132709]
  Large language models (LLMs) can follow natural language instructions with human-level fluency. evaluating LLMs on realistic text generation tasks for healthcare remains challenging. We introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data.
  arXiv  Detail & Related papers  (2023-08-27T12:24:39Z)
• TREEMENT: Interpretable Patient-Trial Matching via Personalized Dynamic Tree-Based Memory Network [54.332862955411656]
  Clinical trials are critical for drug development but often suffer from expensive and inefficient patient recruitment. In recent years, machine learning models have been proposed for speeding up patient recruitment via automatically matching patients with clinical trials. We introduce a dynamic tree-based memory network model named TREEMENT to provide accurate and interpretable patient trial matching.
  arXiv  Detail & Related papers  (2023-07-19T12:35:09Z)
• Self-Verification Improves Few-Shot Clinical Information Extraction [73.6905567014859]
  Large language models (LLMs) have shown the potential to accelerate clinical curation via few-shot in-context learning. They still struggle with issues regarding accuracy and interpretability, especially in mission-critical domains such as health. Here, we explore a general mitigation framework using self-verification, which leverages the LLM to provide provenance for its own extraction and check its own outputs.
  arXiv  Detail & Related papers  (2023-05-30T22:05:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.