Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA

• URL: http://arxiv.org/abs/2604.17316v1
• Date: Sun, 19 Apr 2026 08:11:45 GMT
• Title: Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA
• Abstract summary: Social descriptors of a patient distort uncertainty signals and model accuracy.<n>"Homosexual" markers consistently trigger performance drops, and intersectional identities produce idiosyncratic, non-additive harms to calibration.
• Score: 5.307797621937223
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safe clinical deployment of Large Language Models (LLMs) requires not only high accuracy but also robust uncertainty calibration to ensure models defer to clinicians when appropriate. Our paper investigates how social descriptors of a patient (specifically sexual orientation and religious affiliation) distort these uncertainty signals and model accuracy. Evaluating nine general-purpose and biomedical LLMs on 2,364 medical questions and their counterfactual variants, we demonstrate that identity markers cause a "calibration crisis". "Homosexual" markers consistently trigger performance drops, and intersectional identities produce idiosyncratic, non-additive harms to calibration. Moreover, a clinician-validated case study in an open-ended generation setting confirms that these failures are not an artifact of the multiple-choice format. Our results demonstrate that the presence of social identity cues does not merely shift predictions; it affects the reliability of confidence signals, posing a significant risk to equitable care and safe deployment in confidence-based clinical workflows.

Related papers

• When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information [14.659028787864996]
  Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks.<n>We conduct a behavioral analysis of LLMs under clinical information uncertainty.<n>We observe significant variation across models in their ability to abstain when the correct answer is unavailable.
  arXiv  Detail & Related papers  (2026-08-10T03:28:46Z)
• Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support [1.4678996144709464]
  This Perspective concerns the most consequential of these applications: the autonomous triage of self-presenting, undifferentiated patients.<n>A model optimized to continue the most probable text is not optimized to act safely when the safe answer is the improbable must-not-miss diagnosis.<n>The core deficit is therefore one of information gathering under uncertainty.
  arXiv  Detail & Related papers  (2026-07-29T07:12:46Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure? [33.91781729231922]
  We benchmark 8 methods across 12 vision-language models (VLMs) on clinical visual question-answering (VQA)<n>We find that UE quality is not an intrinsic property of the UE method: it tracks model accuracy, degrading precisely where the model performance is weakest.<n>Our results position UE as a diagnostic tool for identifying fragile predictions and motivate perturbation-based evaluation as a path toward safe clinical deployment.
  arXiv  Detail & Related papers  (2026-06-15T11:27:11Z)
• Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence? [50.17126900886782]
  It remains unclear whether models can apply their own linguistic confidence framework to associate markers with specific confidence levels in a stable and generalizable way.<n>We formalize _marker internal confidence_ (MIC) as the estimated intrinsic confidence a model associates with a specific epistemic marker in a given task domain.<n>Applying our analysis framework to diverse models and tasks, we find that LLMs remain faithfully miscalibrated even under model-centric interpretation of marker meanings.
  arXiv  Detail & Related papers  (2026-05-27T17:38:00Z)
• CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Prediction [10.129412789850239]
  We propose a framework that aligns clinical LM-based risk estimates and uncertainty with both individual error likelihoods and cohort-level ambiguities.<n>Experiments on MIMIC-IV clinical risk prediction tasks show that CURA consistently improves calibration metrics without substantially compromising discrimination.
  arXiv  Detail & Related papers  (2026-04-16T05:58:37Z)
• Multi-Agent Reasoning with Consistency Verification Improves Uncertainty Calibration in Medical MCQA [0.0]
  Multi-agent framework combines domain-specific specialist agents with Two-Phase Verification and S-Score Weighted Fusion.<n>Four specialist agents generate independent diagnoses using Qwen2.5-7B-Instruct.<n>S-scores drive a weighted fusion strategy that selects the final answer and calibrates reported confidence.
  arXiv  Detail & Related papers  (2026-03-25T16:22:53Z)
• Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification [60.18369393468405]
  Existing verifiers usually underperform owing to a lack of domain knowledge and limited calibration.<n>GLEAN compiles expert-curated protocols into trajectory-informed, well-calibrated correctness signals.<n>We empirically validate GLEAN with agentic clinical diagnosis across three diseases from the MIMIC-IV dataset.
  arXiv  Detail & Related papers  (2026-03-03T09:36:43Z)
• Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language Models [87.66870367661342]
  Large language models (LLMs) are used in AI applications in healthcare.<n>Red-teaming framework that continuously stress-test LLMs can reveal significant weaknesses in four safety-critical domains.<n>A suite of adversarial agents is applied to autonomously mutate test cases, identify/evolve unsafe-triggering strategies, and evaluate responses.<n>Our framework delivers an evolvable, scalable, and reliable safeguard for the next generation of medical AI.
  arXiv  Detail & Related papers  (2025-07-30T08:44:22Z)
• Towards Reliable Medical Image Segmentation by Modeling Evidential Calibrated Uncertainty [57.023423137202485]
  Concerns regarding the reliability of medical image segmentation persist among clinicians.<n>We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks.<n>By leveraging subjective logic theory, we explicitly model probability and uncertainty for medical image segmentation.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)
• Bayesian autoencoders with uncertainty quantification: Towards trustworthy anomaly detection [78.24964622317634]
  In this work, the formulation of Bayesian autoencoders (BAEs) is adopted to quantify the total anomaly uncertainty. To evaluate the quality of uncertainty, we consider the task of classifying anomalies with the additional option of rejecting predictions of high uncertainty. Our experiments demonstrate the effectiveness of the BAE and total anomaly uncertainty on a set of benchmark datasets and two real datasets for manufacturing.
  arXiv  Detail & Related papers  (2022-02-25T12:20:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.