CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents

• URL: http://arxiv.org/abs/2605.09675v1
• Date: Sun, 10 May 2026 17:45:01 GMT
• Title: CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents
• Abstract summary: We introduce CodeClinic, a benchmark for evaluating whether LLM agents can synthesize and compose reusable clinical skills.<n>We propose an offline autoformalization pipeline that converts natural-language clinical guidelines into reusable and verified Python skill libraries.
• Score: 17.491888826944074
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Clinical reasoning agents based on large language models (LLMs) aim to automate tasks such as intensive care unit (ICU) monitoring and patient state tracking from electronic health records (EHRs). Existing systems typically rely on manually curated clinical tools or skills for concepts such as sepsis detection and organ failure assessment. However, maintaining these tool libraries requires substantial expert effort, while zero-shot querying or code generation often produces inefficient and unreliable reasoning chains, especially under institution-specific clinical policies. We introduce CodeClinic, a benchmark built on MIMIC-IV for evaluating whether LLM agents can synthesize and compose reusable clinical skills instead of relying on fixed toolboxes. The benchmark contains two complementary tasks: longitudinal ICU surveillance and compositional information seeking. The longitudinal setting simulates monitoring patient trajectories with structured decisions every four hours across 25 findings and eight clinical families, while the compositional setting spans 63k instances across 259 tasks in nine domains and is stratified by compositional dependency depth to evaluate increasingly complex multi-step reasoning. We further propose an offline autoformalization pipeline that converts natural-language clinical guidelines into reusable and verified Python skill libraries through iterative LLM refinement. Compared with zero-shot code generation, the resulting libraries improve consistency while reducing per-query token usage by up to 40%.

Related papers

• Sparse Multi-Stage Expert-Agent Routing for Complex Clinical Reasoning [8.604005197463147]
  We propose Sparse Multi-Stage Expert-Agent, a language-based clinical reasoning framework that models diagnosis as a stage-wise routing process.<n>Given progressively available clinical evidence, the framework maintains an evolving case state and adaptively activates a sparse set of medical expert agents.<n>On reconstructed multi-stage cases from MAC and AgentClinic-NEJM, our framework reduces the average number of activated experts from 17.0 to 3.0 whilst maintaining strong fact-level diagnostic quality.
  arXiv  Detail & Related papers  (2026-08-22T13:23:30Z)
• ResidencyRL: Reinforcement Learning in Simulated Clinical Environments [53.76751756107077]
  We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters.<n>On heldout evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions.<n>Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons.
  arXiv  Detail & Related papers  (2026-08-07T17:04:41Z)
• Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases [71.12461204050985]
  MedSP1000 is an SP-derived interactive benchmark for clinical-agent evaluation.<n>It converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubrics.<n>Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios.
  arXiv  Detail & Related papers  (2026-06-03T17:17:16Z)
• EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs [51.129595320595094]
  Clinical decision-making (CDM) is central to real-world clinical, where clinicians infer diagnoses, select treatments, or anticipate future health outcomes under incomplete evidence.<n>LLM models are increasingly used to support these decisions due to strong language capabilities, broad biomedical knowledge, and efficiency.<n>The reliability of LLMs on real-world clinical decision tasks remains insufficiently understood.
  arXiv  Detail & Related papers  (2026-05-28T22:38:26Z)
• Agentic Cognitive Profiling: Realigning Automated Alzheimer's Disease Detection with Clinical Construct Validity [66.94391219005291]
  This paper proposes Agentic Cognitive Profiling (ACP), an agentic framework that realigns automated screening with clinical protocol logic.<n>Central to our design is decoupling semantic understanding from measurement by delegating all quantification to deterministic function calling.<n>ACP achieves 90.5% score match rate in task examination and 85.3% accuracy in AD prediction, surpassing popular baselines.
  arXiv  Detail & Related papers  (2026-03-18T06:15:35Z)
• Leveraging LLMs for Structured Data Extraction from Unstructured Patient Records [0.0]
  Manual chart review remains an extremely time-consuming and resource-intensive component of clinical research.<n>We present a framework for automated structured feature extraction from clinical notes leveraging locally deployed large language models (LLMs)<n>This framework demonstrates the potential of LLM systems to reduce the burden of manual chart review and increase consistency in data capture.
  arXiv  Detail & Related papers  (2025-12-03T14:10:12Z)
• MedDCR: Learning to Design Agentic Workflows for Medical Coding [55.51674334874892]
  Medical coding converts free-text clinical notes into standardized diagnostic and procedural codes.<n>We present MedDCR, a closed-loop framework that treats design as a learning problem.<n>On benchmark datasets, MedDCR outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-11-17T13:30:51Z)
• Timely Clinical Diagnosis through Active Test Selection [49.091903570068155]
  We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design) to better emulate real-world diagnostic reasoning.<n>LLMs act as flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data.<n>We evaluate ACTMED on real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretability, and resource use.
  arXiv  Detail & Related papers  (2025-10-21T18:10:45Z)
• Toward Reliable Clinical Coding with Language Models: Verification and Lightweight Adaptation [3.952186976672079]
  We show that lightweight interventions, including prompt engineering and small-scale fine-tuning, can improve accuracy without the computational overhead of search-based methods.<n>To address hierarchically near-miss errors, we introduce clinical code verification as both a standalone task and a pipeline component.
  arXiv  Detail & Related papers  (2025-10-08T23:50:58Z)
• EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysis [62.00431604976949]
  EndoBench is the first comprehensive benchmark specifically designed to assess MLLMs across the full spectrum of endoscopic practice.<n>We benchmark 23 state-of-the-art models, including general-purpose, medical-specialized, and proprietary MLLMs.<n>Our experiments reveal: proprietary MLLMs outperform open-source and medical-specialized models overall, but still trail human experts.
  arXiv  Detail & Related papers  (2025-05-29T16:14:34Z)
• MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models [10.46932473088646]
  We introduce MedGUIDE, a new benchmark for evaluating Large Language Models (LLMs) on their ability to make guideline-consistent clinical decisions.<n> MedGUIDE is constructed from 55 curated NCCN decision trees across 17 cancer types.<n>We apply a two-stage quality selection process, combining expert-labeled reward models and LLM-as-a-judge ensembles across ten clinical and linguistic criteria, to select 7,747 high-quality samples.
  arXiv  Detail & Related papers  (2025-05-16T18:21:52Z)
• AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments [2.567146936147657]
  We introduce AgentClinic, a multimodal agent benchmark for evaluating large language models (LLM) in simulated clinical environments.<n>We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy.
  arXiv  Detail & Related papers  (2024-05-13T17:38:53Z)
• Large Language Models in the Clinic: A Comprehensive Benchmark [63.21278434331952]
  We build a benchmark ClinicBench to better understand large language models (LLMs) in the clinic. We first collect eleven existing datasets covering diverse clinical language generation, understanding, and reasoning tasks. We then construct six novel datasets and clinical tasks that are complex but common in real-world practice. We conduct an extensive evaluation of twenty-two LLMs under both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2024-04-25T15:51:06Z)
• Self-Verification Improves Few-Shot Clinical Information Extraction [73.6905567014859]
  Large language models (LLMs) have shown the potential to accelerate clinical curation via few-shot in-context learning. They still struggle with issues regarding accuracy and interpretability, especially in mission-critical domains such as health. Here, we explore a general mitigation framework using self-verification, which leverages the LLM to provide provenance for its own extraction and check its own outputs.
  arXiv  Detail & Related papers  (2023-05-30T22:05:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.