When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information

• URL: http://arxiv.org/abs/2608.09080v1
• Date: Mon, 10 Aug 2026 03:28:46 GMT
• Title: When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information
• Abstract summary: Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.<n>We conduct a behavioral analysis of LLMs under clinical information uncertainty.<n>We observe significant variation across models in their ability to abstain when the correct answer is unavailable.
• Score: 14.659028787864996
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks. However, their reliability under uncertainty remains poorly understood which raises critical concerns for deployment in high-stakes clinical settings. In such environments, incorrect predictions are inherently risky, but confident incorrect predictions can be particularly harmful as they may mislead clinical decision-making. In this paper, we conduct a systematic behavioral analysis of LLMs under clinical information uncertainty. We propose an evaluation framework based on the MedMCQA dataset consisting of two complementary uncertainty settings. First, we introduce linguistic uncertainty cues through prompt modifications to simulate ambiguous clinical contexts. Second, we construct an answer removal setting, wherein the correct option is deliberately excluded mandating the model to recognize insufficient information and abstain. We analyze both model accuracy and confidence behavior using multiple calibration metrics including calibration gap, Expected Calibration Error (ECE), and Unsafe Confident Error Rate (UCER) across 500 medical questions. Our results reveal a consistent failure mode, i.e., although accuracy degrades under increasing uncertainty, model confidence remains misaligned with accuracy. This leads to a substantial increase in unsafe confident errors, indicating that model confidence remains largely insensitive to clinically meaningful information loss. Furthermore, we observe significant variation across models in their ability to abstain when the correct answer is unavailable, with some models persistently producing high confidence hallucinated answers. These findings expose critical limitations in the epistemic reliability of current LLMs and highlight the need for uncertainty aware evaluation methods prior to their deployment in clinical workflows.

Related papers

• Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure? [33.91781729231922]
  We benchmark 8 methods across 12 vision-language models (VLMs) on clinical visual question-answering (VQA)<n>We find that UE quality is not an intrinsic property of the UE method: it tracks model accuracy, degrading precisely where the model performance is weakest.<n>Our results position UE as a diagnostic tool for identifying fragile predictions and motivate perturbation-based evaluation as a path toward safe clinical deployment.
  arXiv  Detail & Related papers  (2026-06-15T11:27:11Z)
• Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA [5.307797621937223]
  Social descriptors of a patient distort uncertainty signals and model accuracy.<n>"Homosexual" markers consistently trigger performance drops, and intersectional identities produce idiosyncratic, non-additive harms to calibration.
  arXiv  Detail & Related papers  (2026-04-19T08:11:45Z)
• CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Prediction [10.129412789850239]
  We propose a framework that aligns clinical LM-based risk estimates and uncertainty with both individual error likelihoods and cohort-level ambiguities.<n>Experiments on MIMIC-IV clinical risk prediction tasks show that CURA consistently improves calibration metrics without substantially compromising discrimination.
  arXiv  Detail & Related papers  (2026-04-16T05:58:37Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• Calibrated Bayesian Deep Learning for Explainable Decision Support Systems Based on Medical Imaging [6.826979426009301]
  It is imperative that models quantify uncertainty in a manner that correlates with prediction correctness, allowing clinicians to identify unreliable outputs for further review.<n>The present paper proposes a generalizable probabilistic optimization framework grounded in Bayesian deep learning.<n>Specifically, a novel Confidence-Uncertainty Boundary Loss (CUB-Loss) is introduced that imposes penalties on high-certainty errors and low-certainty correct predictions.<n>The proposed framework is validated on three distinct medical imaging tasks: automatic screening of pneumonia, diabetic retinopathy detection, and identification of skin lesions.
  arXiv  Detail & Related papers  (2026-02-12T14:03:41Z)
• Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations [49.84786015324238]
  Confidence estimation (CE) indicates how reliable the answers of large language models (LLMs) are, and can impact user trust and decision-making.<n>We present a comprehensive evaluation framework for CE that measures their confidence quality on three new aspects.<n>These include robustness of confidence against prompt perturbations, stability across semantic equivalent answers, and sensitivity to semantically different answers.
  arXiv  Detail & Related papers  (2026-01-12T23:16:50Z)
• MedBayes-Lite: Bayesian Uncertainty Quantification for Safe Clinical Decision Support [5.22077647816926]
  We propose MedBayes-Lite, a lightweight Bayesian enhancement for transformer-based clinical language models.<n> MedBayes-Lite embeds uncertainty quantification directly into existing transformer pipelines without any retraining or architectural rewiring.<n>It consistently improves calibration and trustworthiness, reducing overconfidence by 32 to 48 percent.
  arXiv  Detail & Related papers  (2025-11-20T18:33:12Z)
• Towards Reliable LLM-based Robot Planning via Combined Uncertainty Estimation [68.106428321492]
  Large language models (LLMs) demonstrate advanced reasoning abilities, enabling robots to understand natural language instructions and generate high-level plans with appropriate grounding.<n>LLMs hallucinations present a significant challenge, often leading to overconfident yet potentially misaligned or unsafe plans.<n>We present Combined Uncertainty estimation for Reliable Embodied planning (CURE), which decomposes the uncertainty into epistemic and intrinsic uncertainty, each estimated separately.
  arXiv  Detail & Related papers  (2025-10-09T10:26:58Z)
• Probabilistic Modeling of Disparity Uncertainty for Robust and Efficient Stereo Matching [61.73532883992135]
  We propose a new uncertainty-aware stereo matching framework.<n>We adopt Bayes risk as the measurement of uncertainty and use it to separately estimate data and model uncertainty.
  arXiv  Detail & Related papers  (2024-12-24T23:28:20Z)
• Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations [63.330182403615886]
  A major barrier towards the practical deployment of large language models (LLMs) is their lack of reliability. Three situations where this is particularly apparent are correctness, hallucinations when given unanswerable questions, and safety. In all three cases, models should ideally abstain from responding, much like humans, whose ability to understand uncertainty makes us refrain from answering questions we don't know.
  arXiv  Detail & Related papers  (2024-04-16T23:56:38Z)
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [69.83976050879318]
  In large language models (LLMs), identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions.
  arXiv  Detail & Related papers  (2023-11-15T05:58:35Z)
• Towards Reliable Medical Image Segmentation by Modeling Evidential Calibrated Uncertainty [57.023423137202485]
  Concerns regarding the reliability of medical image segmentation persist among clinicians.<n>We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks.<n>By leveraging subjective logic theory, we explicitly model probability and uncertainty for medical image segmentation.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.