LiveClin: A Live Clinical Benchmark without Leakage

• URL: http://arxiv.org/abs/2602.16747v1
• Date: Wed, 18 Feb 2026 03:59:46 GMT
• Title: LiveClin: A Live Clinical Benchmark without Leakage
• Abstract summary: LiveClin is a live benchmark designed for approximating real-world clinical practice.<n>We transform authentic patient cases into complex, multimodal evaluation scenarios that span the entire clinical pathway.<n>Our evaluation of 26 models on LiveClin reveals the profound difficulty of these real-world scenarios, with the top-performing model achieving a Case Accuracy of just 35.7%.
• Score: 50.45415584327275
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The reliability of medical LLM evaluation is critically undermined by data contamination and knowledge obsolescence, leading to inflated scores on static benchmarks. To address these challenges, we introduce LiveClin, a live benchmark designed for approximating real-world clinical practice. Built from contemporary, peer-reviewed case reports and updated biannually, LiveClin ensures clinical currency and resists data contamination. Using a verified AI-human workflow involving 239 physicians, we transform authentic patient cases into complex, multimodal evaluation scenarios that span the entire clinical pathway. The benchmark currently comprises 1,407 case reports and 6,605 questions. Our evaluation of 26 models on LiveClin reveals the profound difficulty of these real-world scenarios, with the top-performing model achieving a Case Accuracy of just 35.7%. In benchmarking against human experts, Chief Physicians achieved the highest accuracy, followed closely by Attending Physicians, with both surpassing most models. LiveClin thus provides a continuously evolving, clinically grounded framework to guide the development of medical LLMs towards closing this gap and achieving greater reliability and real-world utility. Our data and code are publicly available at https://github.com/AQ-MedAI/LiveClin.

Related papers

• ResidencyRL: Reinforcement Learning in Simulated Clinical Environments [53.76751756107077]
  We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters.<n>On heldout evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions.<n>Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons.
  arXiv  Detail & Related papers  (2026-08-07T17:04:41Z)
• Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases [71.12461204050985]
  MedSP1000 is an SP-derived interactive benchmark for clinical-agent evaluation.<n>It converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubrics.<n>Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios.
  arXiv  Detail & Related papers  (2026-06-03T17:17:16Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models [8.050059911826338]
  Large Language Models (LLMs) excel on standardized medical exams, but high scores often fail to translate to high-quality responses for real-world medical queries.<n>We introduce QuarkMedBench, an ecologically valid benchmark tailored for real-world medical LLM assessment.
  arXiv  Detail & Related papers  (2026-03-14T01:51:43Z)
• MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences [50.71326426975699]
  We present MedArena, an interactive evaluation platform for medical large language models (LLMs)<n>Given a clinician-provided query, MedArena presents responses from two randomly selected models and asks the user to select the preferred response.<n>Out of 1571 preferences collected across 12 LLMs up to November 1, 2025, Gemini 2.0 Flash Thinking, Gemini 2.5 Pro, and GPT-4o were the top three models by Bradley-Terry rating.
  arXiv  Detail & Related papers  (2026-03-13T22:30:26Z)
• ClinConsensus: A Consensus-Based Benchmark for Evaluating Chinese Medical LLMs across Difficulty Levels [39.33170904610862]
  Large language models (LLMs) are increasingly applied to health management, showing promise across disease prevention, clinical decision-making, and long-term care.<n>We introduce ClinConsensus, a Chinese medical benchmark curated, validated and quality-controlled by clinical experts.<n> ClinConsensus comprises 2500 open-ended cases spanning the full continuum of care--from prevention and intervention to long-term follow-up--covering 36 medical specialties, 12 common clinical task types, and progressively increasing levels of complexity.
  arXiv  Detail & Related papers  (2026-03-02T17:17:18Z)
• LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluation [22.211535340726073]
  LiveMedBench is a continuously updated benchmark that harvests real-world clinical cases from online medical communities.<n>LiveMedBench comprises 2,756 real-world cases spanning 38 medical specialties and multiple languages, paired with 16,702 unique evaluation criteria.<n>Extensive evaluation reveals that even the best-performing model achieves only 39.2%, and 84% of models exhibit performance degradation on post-cutoff cases.
  arXiv  Detail & Related papers  (2026-02-10T23:38:25Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• A Real-World Evaluation of LLM Medication Safety Reviews in NHS Primary Care [5.167350493769989]
  This is the first evaluation of an LLM-based medication safety review system on real NHS primary care data.<n>We strategically sampled patients to capture a broad range of clinical complexity and medication safety risk.<n>Our primary LLM system showed strong performance in recognising when a clinical issue is present.
  arXiv  Detail & Related papers  (2025-12-24T11:58:49Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language Models [87.66870367661342]
  Large language models (LLMs) are used in AI applications in healthcare.<n>Red-teaming framework that continuously stress-test LLMs can reveal significant weaknesses in four safety-critical domains.<n>A suite of adversarial agents is applied to autonomously mutate test cases, identify/evolve unsafe-triggering strategies, and evaluate responses.<n>Our framework delivers an evolvable, scalable, and reliable safeguard for the next generation of medical AI.
  arXiv  Detail & Related papers  (2025-07-30T08:44:22Z)
• Structured Outputs Enable General-Purpose LLMs to be Medical Experts [50.02627258858336]
  Large language models (LLMs) often struggle with open-ended medical questions.<n>We propose a novel approach utilizing structured medical reasoning.<n>Our approach achieves the highest Factuality Score of 85.8, surpassing fine-tuned models.
  arXiv  Detail & Related papers  (2025-03-05T05:24:55Z)
• PRISM: Patient Records Interpretation for Semantic Clinical Trial Matching using Large Language Models [4.438101430231511]
  We present the first, end-to-end large-scale empirical evaluation of clinical trial matching using real-world EHRs. Our study showcases the capability of LLMs to accurately match patients with appropriate clinical trials.
  arXiv  Detail & Related papers  (2024-04-23T22:33:19Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• LongHealth: A Question Answering Benchmark with Long Clinical Documents [36.05587855811346]
  We present the LongHealth benchmark, comprising 20 detailed fictional patient cases across various diseases. The benchmark challenges LLMs with 400 multiple-choice questions in three categories: information extraction, negation, and sorting. We evaluated nine open-source LLMs with a minimum of 16,000 tokens and also included OpenAI's proprietary and cost-efficient GPT-3.5 Turbo for comparison.
  arXiv  Detail & Related papers  (2024-01-25T19:57:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.