RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation

• URL: http://arxiv.org/abs/2605.13542v1
• Date: Wed, 13 May 2026 13:52:42 GMT
• Title: RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation
• Abstract summary: RealICU is a hindsight-annotated benchmark for evaluating large language models under realistic ICU conditions.<n>We release two datasets: RealICU-Gold with 930-window annotations from 94 MIMIC-IV patients, and RealICU-Scale with 11,862 windows extended by Oracle.
• Score: 46.82418087865201
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Intensive care units (ICU) generate long, dense and evolving streams of clinical information, where physicians must repeatedly reassess patient states under time pressure, underscoring a clear need for reliable AI decision support. Existing ICU benchmarks typically treat historical clinician actions as ground truth. However, these actions are made under incomplete information and limited temporal context of the underlying patient state, and may therefore be suboptimal, making it difficult to assess the true reasoning capabilities of AI systems. We introduce RealICU, a hindsight-annotated benchmark for evaluating large language models (LLMs) under realistic ICU conditions, where labels are created after senior physicians review the full patient trajectory. We formulate four physician-motivated tasks: assess Patient Status, Acute Problems, Recommended Actions, and Red Flag actions that risk unsafe outcomes. We partition each trajectory with 30-min windows and release two datasets: RealICU-Gold with 930-window annotations from 94 MIMIC-IV patients, and RealICU-Scale with 11,862 windows extended by Oracle, a physician-validated LLM hindsight labeler. Existing LLMs including memory-augmented ones performed poorly on RealICU, exposing two failure modes: a recall-safety tradeoff for clinical recommendations, and an anchoring bias to early interpretations of the patient. We further introduce ICU-Evo to study structured-memory agents that improves long-horizon reasoning but does not fully eliminate safety failures. Together, RealICU provides a clinically grounded testbed for measuring and improving AI sequential decision-support in high-stakes care. Project page: https://chengzhi-leo.github.io/RealICU-Bench/

Related papers

• KEPIL: Knowledge-Enhanced Prompt-Image Learning for Prompt-Robust Disease Detection [6.447908430647854]
  radiological findings are long-tailed in practice, leaving some conditions underrepresented and making zero-shot inference essential.<n>We present textitKEPIL, a prompt-robust framework that integrates curated medical knowledge to stabilize zero-shot generalization.
  arXiv  Detail & Related papers  (2026-05-09T19:29:01Z)
• Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations [2.337503919179969]
  We study challenging patient behaviors that commonly arise in real medical consultations.<n>For each behavior, we specify concrete failure criteria that capture unsafe responses.<n>We evaluate a range of open- and closed-source LLMs on their responses to challenging patient utterances.
  arXiv  Detail & Related papers  (2026-03-31T07:42:07Z)
• Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Models [48.95516224614331]
  We introduce MedGaze-Bench, the first benchmark leveraging clinician gaze as a Cognitive Cursor to assess intent understanding across surgery, emergency simulation, and diagnostic interpretation.<n>Our benchmark addresses three fundamental challenges: visual homogeneity of anatomical structures, strict temporal-causal dependencies in clinical, and implicit adherence to safety protocols.
  arXiv  Detail & Related papers  (2026-01-11T02:20:40Z)
• MedForget: Hierarchy-Aware Multimodal Unlearning Testbed for Medical AI [66.0701326117134]
  MedForget is a hierarchy-aware multimodal unlearning testbed for building compliant medical AI systems.<n>We show that existing methods struggle to achieve complete, hierarchy-aware forgetting without reducing diagnostic performance.<n>We introduce a reconstruction attack that progressively adds hierarchical level context to prompts.
  arXiv  Detail & Related papers  (2025-12-10T17:55:06Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases [48.87360916431396]
  We introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references.<n>We propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey.<n>Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc.
  arXiv  Detail & Related papers  (2025-03-06T18:35:39Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• Detecting Visual Cues in the Intensive Care Unit and Association with Patient Clinical Status [0.9867627975175174]
  Existing patient assessments in the ICU are mostly sporadic and administered manually. We developed a new "masked loss computation" technique that addresses the data imbalance problem. We performed AU inference on 634,054 frames to evaluate the association between facial AUs and clinically important patient conditions.
  arXiv  Detail & Related papers  (2023-11-01T15:07:03Z)
• Predicting Unplanned Readmissions in the Intensive Care Unit: A Multimodality Evaluation [2.2559617939136505]
  A hospital readmission is when a patient who was discharged from the hospital is admitted again for the same or related care within a certain period. We use state-of-the-art machine learning approaches in time-series analysis and natural language processing to predict Unplanned Readmissions in ICUs.
  arXiv  Detail & Related papers  (2023-05-14T12:20:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.