Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

• URL: http://arxiv.org/abs/2609.00296v1
• Date: Mon, 31 Aug 2026 19:41:58 GMT
• Title: Toward Workflow-Aware Benchmarking for Healthcare NLP Agents
• Abstract summary: We introduce an episode-level evaluation protocol for healthcare NLP agents.<n>It separates evidence across model, agent, and simulated-workflow behavior.<n>It defines annotation and scoring for state continuity, evidence traceability, and escalation decisions.
• Score: 4.287745257410248
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM) agents are increasingly proposed for healthcare tasks such as clinical documentation, evidence retrieval, patient messaging, and care coordination. Yet many evaluations remain limited to static medical question answering or one-shot generation, under-representing longitudinal state, interruptions, and human handoffs. We introduce an episode-level evaluation protocol for healthcare NLP agents. The protocol separates evidence across model, agent, and simulated-workflow behavior; specifies a five-field episode schema; and defines annotation and scoring for state continuity, evidence traceability, and escalation decisions. It is instantiated as four task templates: documentation update, evidence retrieval, patient messaging, and triage handoff. The protocol does not claim to measure clinical outcomes or deployment value. Instead, it supplies a reproducible intermediate evaluation layer between static benchmarks and prospective workflow studies, with an explicit cost-sensitive treatment of missed versus unnecessary escalation.

Related papers

• Toward Automated Detection of Documentation Inconsistencies in Electronic Health Records [16.68744679025527]
  General-domain large language model (LLM) can surface from real-world discharge summaries.<n>LLM surfaced 3,460 candidate inconsistencies, affecting 69.7% of admissions.
  arXiv  Detail & Related papers  (2026-07-24T23:43:35Z)
• ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflows [7.27267618572654]
  ProtoMedAgent formalizes multimodal clinical reporting as an iterative, zero-gradient test-time optimization problem.<n>Online generation is strictly constrained by exact set-theoretic differentials and a reflective Scribe-Critic loop.<n>ProtoMedAgent achieves 91.2% Comparison Set Faithfulness where it fundamentally outperforms standard RAG (46.2%)
  arXiv  Detail & Related papers  (2026-05-13T20:57:37Z)
• ADAPTS: Agentic Decomposition for Automated Protocol-agnostic Tracking of Symptoms [36.94429692322632]
  ADAPTS (Agentic Decomposition for Automated Protocol-agnostic Tracking of Symptoms) is a framework for automated rating of depression and anxiety severity.<n>This approach decomposes long-form clinical interviews into symptom-specific reasoning tasks.<n>Ratings approximated expert benchmarks more closely than original human ratings.
  arXiv  Detail & Related papers  (2026-05-04T23:08:42Z)
• Agentic Cognitive Profiling: Realigning Automated Alzheimer's Disease Detection with Clinical Construct Validity [66.94391219005291]
  This paper proposes Agentic Cognitive Profiling (ACP), an agentic framework that realigns automated screening with clinical protocol logic.<n>Central to our design is decoupling semantic understanding from measurement by delegating all quantification to deterministic function calling.<n>ACP achieves 90.5% score match rate in task examination and 85.3% accuracy in AD prediction, surpassing popular baselines.
  arXiv  Detail & Related papers  (2026-03-18T06:15:35Z)
• AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning [73.50200033931148]
  We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists.<n>By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback.<n> Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations.
  arXiv  Detail & Related papers  (2026-01-23T11:59:13Z)
• MedDCR: Learning to Design Agentic Workflows for Medical Coding [55.51674334874892]
  Medical coding converts free-text clinical notes into standardized diagnostic and procedural codes.<n>We present MedDCR, a closed-loop framework that treats design as a learning problem.<n>On benchmark datasets, MedDCR outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-11-17T13:30:51Z)
• Evaluating Prompting Strategies with MedGemma for Medical Order Extraction [6.312830352605384]
  This paper details our team submission to the MEDIQA-OE-2025 Shared Task.<n>We investigate the performance of MedGemma, a new domain-specific open-source language model, for structured order extraction.
  arXiv  Detail & Related papers  (2025-11-13T18:22:32Z)
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services [49.70819009392778]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) offer opportunities to augment dispatchers.<n>This study aimed to develop and evaluate a taxonomy-grounded, multi-agent system for simulating realistic scenarios.
  arXiv  Detail & Related papers  (2025-10-24T08:01:21Z)
• Automated Clinical Problem Detection from SOAP Notes using a Collaborative Multi-Agent LLM Architecture [8.072932739333309]
  We introduce a collaborative multi-agent system (MAS) that models a clinical consultation team to address this gap.<n>The system is tasked with identifying clinical problems by analyzing only the Subjective (S) and Objective (O) sections of SOAP notes.<n>A Manager agent orchestrates a dynamically assigned team of specialist agents who engage in a hierarchical, iterative debate to reach a consensus.
  arXiv  Detail & Related papers  (2025-08-29T17:31:24Z)
• DENSE: Longitudinal Progress Note Generation with Temporal Modeling of Heterogeneous Clinical Notes Across Hospital Visits [0.1578515540930834]
  Progress notes are among the most clinically meaningful artifacts in an Electronic Health Record.<n>Despite their importance, they are underrepresented in large-scale EHR datasets.<n>We present DENSE, a system designed to align with clinical documentation by simulating how physicians reference past encounters.
  arXiv  Detail & Related papers  (2025-07-18T17:00:27Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)
• Learning Contextualized Document Representations for Healthcare Answer Retrieval [68.02029435111193]
  Contextual Discourse Vectors (CDV) is a distributed document representation for efficient answer retrieval from long documents. Our model leverages a dual encoder architecture with hierarchical LSTM layers and multi-task training to encode the position of clinical entities and aspects alongside the document discourse. We show that our generalized model significantly outperforms several state-of-the-art baselines for healthcare passage ranking.
  arXiv  Detail & Related papers  (2020-02-03T15:47:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.