LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making

• URL: http://arxiv.org/abs/2607.09322v2
• Date: Mon, 13 Jul 2026 07:55:03 GMT
• Title: LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making
• Abstract summary: LongMedBench is a real-world EHR-based benchmark for long-horizon clinical decision-making.<n>It comprises 335 patients, with 19.72 inpatient visits per patient on average and 44.91 medical events per visit.<n>Our experiments show that while recent LLMs can make good use of explicit timestamps, they have challenges in implicit time inference.
• Score: 24.67108801971425
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this work, we introduce LongMedBench, a real-world EHR-based benchmark for long-horizon clinical decision-making. Prior evaluations of LLM-based medical agents have largely emphasized short-context knowledge QA and tool use. However, real-world medical care is inherently longitudinal, and clinicians must aggregate evidence across repeated visits, tests, and evolving treatments. Therefore, long-horizon interaction is essential for realistic assessment. LongMedBench is constructed via a reproducible pipeline that integrates MIMIC-IV admission records and clinical notes into time-series event streams and long-context memory datasets, enabling long-horizon, multi-session interactions between agents and a clinical environment. It comprises 335 patients, with 19.72 inpatient visits per patient on average and 44.91 medical events per visit. Guided by the long-horizon decision process, we propose an evaluation taxonomy with three suites: fact-based QA, temporal reasoning, and long-horizon decision-making. This taxonomy measures how agents understand and leverage historical patient information over extended horizons. Our experiments show that while recent LLMs can make good use of explicit timestamps, they have challenges in implicit time inference; The RAG and agent memory system can improve the performance of information retrieval tasks, but the performance of decision-making tasks is highly dependent on the model's immediate context.

Related papers

• MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agents [7.072210444087386]
  We propose Medical Structured Multimodal Memory (MSM-Mem), an agentic memory framework that enables medical AI agents to evolve through accumulated clinical experiences.<n>MSM-Mem organizes heterogeneous clinical experiences into semantic, episodic, and visual memory and incrementally updates them during inference, allowing the agent to retrieve prior experiences to inform current reasoning.
  arXiv  Detail & Related papers  (2026-08-22T07:17:10Z)
• LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA [23.762697050770157]
  We propose LoMeVQA, a benchmark consisting of 206K longitudinal visual question answering (VQA) pairs for temporal medical image analysis.<n>LoMeVQA covers five tasks: progress classification, progress description, progress report generation, differential region grounding, and differential region description.<n>We introduce MedLong-8B, which achieves state-of-the-art performance across all tasks.
  arXiv  Detail & Related papers  (2026-07-30T07:45:58Z)
• MedGym:A Unified Continuous-Time Benchmark for Dynamic Medical Treatment Reinforcement Learning [16.00709468365385]
  We introduce MedGym, a benchmark environment for dynamic treatment recommendation.<n>MedGym models longitudinal patient evolution in a continuous-time framework.<n>It constructs a medical RL benchmark from clinical data by using Physics-Informed Neural Networks.
  arXiv  Detail & Related papers  (2026-05-31T05:36:03Z)
• EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs [51.129595320595094]
  Clinical decision-making (CDM) is central to real-world clinical, where clinicians infer diagnoses, select treatments, or anticipate future health outcomes under incomplete evidence.<n>LLM models are increasingly used to support these decisions due to strong language capabilities, broad biomedical knowledge, and efficiency.<n>The reliability of LLMs on real-world clinical decision tasks remains insufficiently understood.
  arXiv  Detail & Related papers  (2026-05-28T22:38:26Z)
• Synthesis and Evaluation of Long-term History-aware Medical Dialogue [6.317682711402412]
  We introduce a framework for synthesizing high-quality, long-term medical dialogues with LLMs.<n>We establish three benchmark tasks-In-dialogue Reasoning, Cross-dialogue Reasoning, and Synthesis Reasoning-to evaluate the memory capabilities of healthcare agents.<n> Benchmark experiments show that even state-of-the-art LLMs struggle with MediLongChat.
  arXiv  Detail & Related papers  (2026-05-19T12:38:41Z)
• MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences [50.71326426975699]
  We present MedArena, an interactive evaluation platform for medical large language models (LLMs)<n>Given a clinician-provided query, MedArena presents responses from two randomly selected models and asks the user to select the preferred response.<n>Out of 1571 preferences collected across 12 LLMs up to November 1, 2025, Gemini 2.0 Flash Thinking, Gemini 2.5 Pro, and GPT-4o were the top three models by Bradley-Terry rating.
  arXiv  Detail & Related papers  (2026-03-13T22:30:26Z)
• MedKGent: A Large Language Model Agent Framework for Constructing Temporally Evolving Medical Knowledge Graph [57.54231831309079]
  We introduce MedKGent, a framework for constructing temporally evolving medical Knowledge Graphs.<n>We simulate the emergence of biomedical knowledge via a fine-grained daily time series.<n>The resulting KG contains 156,275 entities and 2,971,384 relational triples.
  arXiv  Detail & Related papers  (2025-08-17T15:14:03Z)
• ER-REASON: A Benchmark Dataset for LLM-Based Clinical Reasoning in the Emergency Room [6.910389029249664]
  Large language models (LLMs) have been extensively evaluated on medical question answering tasks based on licensing exams.<n> ER-Reason is a benchmark designed to evaluate LLM-based clinical reasoning and decision-making in the emergency room.
  arXiv  Detail & Related papers  (2025-05-28T22:43:44Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• Clairvoyance: A Pipeline Toolkit for Medical Time Series [95.22483029602921]
  Time-series learning is the bread and butter of data-driven *clinical decision support* Clairvoyance proposes a unified, end-to-end, autoML-friendly pipeline that serves as a software toolkit. Clairvoyance is the first to demonstrate viability of a comprehensive and automatable pipeline for clinical time-series ML.
  arXiv  Detail & Related papers  (2023-10-28T12:08:03Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.