ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making

• URL: http://arxiv.org/abs/2602.22771v1
• Date: Thu, 26 Feb 2026 09:03:41 GMT
• Title: ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making
• Abstract summary: We develop ClinDet-Bench, a benchmark for evaluating determinability under incomplete information.<n>We find that recent large language models (LLMs) fail to identify determinability under incomplete information.
• Score: 38.53727520114093
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinical decisions are often required under incomplete information. Clinical experts must identify whether available information is sufficient for judgment, as both premature conclusion and unnecessary abstention can compromise patient safety. To evaluate this capability of large language models (LLMs), we developed ClinDet-Bench, a benchmark based on clinical scoring systems that decomposes incomplete-information scenarios into determinable and undeterminable conditions. Identifying determinability requires considering all hypotheses about missing information, including unlikely ones, and verifying whether the conclusion holds across them. We find that recent LLMs fail to identify determinability under incomplete information, producing both premature judgments and excessive abstention, despite correctly explaining the underlying scoring knowledge and performing well under complete information. These findings suggest that existing benchmarks are insufficient to evaluate the safety of LLMs in clinical settings. ClinDet-Bench provides a framework for evaluating determinability recognition, leading to appropriate abstention, with potential applicability to medicine and other high-stakes domains, and is publicly available.

Related papers

• When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information [14.659028787864996]
  Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.<n>We conduct a behavioral analysis of LLMs under clinical information uncertainty.<n>We observe significant variation across models in their ability to abstain when the correct answer is unavailable.
  arXiv  Detail & Related papers  (2026-08-10T03:28:46Z)
• Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support [1.4678996144709464]
  This Perspective concerns the most consequential of these applications: the autonomous triage of self-presenting, undifferentiated patients.<n>A model optimized to continue the most probable text is not optimized to act safely when the safe answer is the improbable must-not-miss diagnosis.<n>The core deficit is therefore one of information gathering under uncertainty.
  arXiv  Detail & Related papers  (2026-07-29T07:12:46Z)
• EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs [51.129595320595094]
  Clinical decision-making (CDM) is central to real-world clinical, where clinicians infer diagnoses, select treatments, or anticipate future health outcomes under incomplete evidence.<n>LLM models are increasingly used to support these decisions due to strong language capabilities, broad biomedical knowledge, and efficiency.<n>The reliability of LLMs on real-world clinical decision tasks remains insufficiently understood.
  arXiv  Detail & Related papers  (2026-05-28T22:38:26Z)
• Medical Model Synthesis Architectures: A Case Study [72.46211022258122]
  Current AI systems struggle with calibrated reasoning under uncertainty.<n>We propose a framework for AI systems that can make practically useful but formally transparent clinical predictions under uncertainty.<n>We show how an initial proof-of-concept of this framework can be used for differential diagnosis.
  arXiv  Detail & Related papers  (2026-05-10T19:30:16Z)
• Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry [2.60905622443275]
  Large language models (LLMs) are increasingly used for decision support in healthcare.<n>Clinical evidence is often incomplete or evolving.<n>We introduce Safe-Psych, a benchmark for evaluating how LLMs handle evolving diagnostic uncertainty.
  arXiv  Detail & Related papers  (2026-05-06T08:52:45Z)
• CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Prediction [10.129412789850239]
  We propose a framework that aligns clinical LM-based risk estimates and uncertainty with both individual error likelihoods and cohort-level ambiguities.<n>Experiments on MIMIC-IV clinical risk prediction tasks show that CURA consistently improves calibration metrics without substantially compromising discrimination.
  arXiv  Detail & Related papers  (2026-04-16T05:58:37Z)
• Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification [60.18369393468405]
  Existing verifiers usually underperform owing to a lack of domain knowledge and limited calibration.<n>GLEAN compiles expert-curated protocols into trajectory-informed, well-calibrated correctness signals.<n>We empirically validate GLEAN with agentic clinical diagnosis across three diseases from the MIMIC-IV dataset.
  arXiv  Detail & Related papers  (2026-03-03T09:36:43Z)
• Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty [7.985656420275706]
  We introduce MedAbstain, a unified benchmark and evaluation protocol for abstention in medical multiple-choice question answering.<n>We show that even state-of-the-art, high-accuracy models often fail to abstain with uncertain.
  arXiv  Detail & Related papers  (2026-01-18T16:19:29Z)
• ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning [58.01333341218153]
  We propose ClinDEF, a dynamic framework for assessing clinical reasoning in LLMs through simulated diagnostic dialogues.<n>Our method generates patient cases and facilitates multi-turn interactions between an LLM-based doctor and an automated patient agent.<n>Experiments show that ClinDEF effectively exposes critical clinical reasoning gaps in state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-12-29T12:58:58Z)
• Timely Clinical Diagnosis through Active Test Selection [49.091903570068155]
  We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design) to better emulate real-world diagnostic reasoning.<n>LLMs act as flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data.<n>We evaluate ACTMED on real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretability, and resource use.
  arXiv  Detail & Related papers  (2025-10-21T18:10:45Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• Evaluating LLMs in Medicine: A Call for Rigor, Transparency [2.2445597370194834]
  Methods: Widely-used benchmark datasets, including MedQA, MedMCQA, PubMedQA, and MMLU, were reviewed for their rigor, transparency, and relevance to clinical scenarios.<n>Alternatives, such as challenge questions in medical journals, were also analyzed to identify their potential as unbiased evaluation tools.
  arXiv  Detail & Related papers  (2025-07-11T16:09:25Z)
• Med-CoDE: Medical Critique based Disagreement Evaluation Framework [72.42301910238861]
  The reliability and accuracy of large language models (LLMs) in medical contexts remain critical concerns.<n>Current evaluation methods often lack robustness and fail to provide a comprehensive assessment of LLM performance.<n>We propose Med-CoDE, a specifically designed evaluation framework for medical LLMs to address these challenges.
  arXiv  Detail & Related papers  (2025-04-21T16:51:11Z)
• LLMs for clinical risk prediction [0.0]
  GPT-4 exhibited significant deficiencies in identifying positive cases and struggled to provide reliable probability estimates for delirium risk. Clinalytix Medical AI demonstrated superior accuracy.
  arXiv  Detail & Related papers  (2024-09-16T11:34:40Z)
• CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making [16.310913127940857]
  We introduce CliBench, a novel benchmark developed from the MIMIC IV dataset. This benchmark offers a comprehensive and realistic assessment of LLMs' capabilities in clinical diagnosis. We conduct a zero-shot evaluation of leading LLMs to assess their proficiency in clinical decision-making.
  arXiv  Detail & Related papers  (2024-06-14T11:10:17Z)
• Towards Reliable Medical Image Segmentation by Modeling Evidential Calibrated Uncertainty [57.023423137202485]
  Concerns regarding the reliability of medical image segmentation persist among clinicians.<n>We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks.<n>By leveraging subjective logic theory, we explicitly model probability and uncertainty for medical image segmentation.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.