Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring

• URL: http://arxiv.org/abs/2605.16386v1
• Date: Mon, 11 May 2026 15:37:24 GMT
• Title: Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring
• Abstract summary: Multimodal large language models (LLMs) are increasingly explored as automated evaluators in clinical settings.<n>We benchmark three frontier LLM families against supervised deep learning models for scoring Clock Drawing Test (CDT) images.<n>Our findings extend the LLM-as-a-judge bias literature from NLP evaluation to clinical assessment.
• Score: 2.835587889687748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models (LLMs) are increasingly explored as automated evaluators in clinical settings, yet their scoring behavior on ordinal clinical scales remains poorly understood. We benchmark three frontier LLM families against supervised deep learning models for scoring Clock Drawing Test (CDT) images on two public datasets using the Shulman rubric. While fully fine-tuned Vision Transformers achieve the best calibration (MAE 0.52, within-1 accuracy 91%), zero-shot LLMs remain competitive on tolerance-based agreement (GPT-5 MAE 0.67, within-1 accuracy 92%) despite higher absolute error. However, per-score analysis reveals that all three LLM families exhibit a pronounced central tendency effect (systematic endpoint compression): predictions are systematically compressed toward the middle of the scale, with over-prediction at the low end (score 0 to 1) and under-prediction at the high end (score 5 to 4). This effect disproportionately affects the clinically critical extremes where accurate scoring most impacts screening decisions for cognitive impairment. Targeted ablations show that neither few-shot exemplars spanning the full score range nor removing clinical terminology from the prompt eliminates the effect. Our findings extend the LLM-as-a-judge bias literature from NLP evaluation to clinical assessment, and highlight the need for calibration-aware evaluation and post-hoc calibration before deploying LLM-based raters in high-stakes screening workflows.

Related papers

• Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking [27.829988205900055]
  We introduce MedQADE, the first standardised open-response clinical benchmark for German.<n>The top-performing evaluator model, Gemini 3 Flash, reached alignment consistent with the physician ceiling.<n>Despite this statistical alignment, automated evaluators exhibited near-absent clinical metacognition.
  arXiv  Detail & Related papers  (2026-07-01T15:55:31Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• TRIAGE: Dialectical Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series with LLMs [38.333764111610044]
  We propose TRIAGE, a framework that trains an LLM to generate dialectical reasoning over competing clinical outcomes.<n> evaluated on three ISMTS benchmarks, TRIAGE achieves an average AUPRC improvement of 3.3% and reduces calibration error by 81%.
  arXiv  Detail & Related papers  (2026-06-08T04:53:44Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• Tipping the Balance: Impact of Class Imbalance Correction on the Performance of Clinical Risk Prediction Models [2.2534253247996214]
  Class-imbalance correction techniques are commonly applied to improve model performance in settings with rare outcomes.<n>This study evaluated the effect of widely used resampling strategies on both discrimination and calibration across real-world clinical prediction tasks.
  arXiv  Detail & Related papers  (2026-02-27T12:29:16Z)
• PREBA: Surgical Duration Prediction via PCA-Weighted Retrieval-Augmented LLMs and Bayesian Averaging Aggregation [51.96735866702332]
  PREBA is a retrieval-augmented framework that integrates PCA-weighted retrieval and Bayesian averaging aggregation.<n> PREBA significantly improves performance-for instance, reducing MAE by up to 40% and raising R2 from -0.13 to 0.62 over zero-shot inference.
  arXiv  Detail & Related papers  (2026-02-27T07:19:23Z)
• Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering [94.37535002230504]
  We develop a training-free, inference-time control framework termed Semantically Decoupled Latent Steering.<n>Our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition.<n>We show that our approach significantly reduces the probability of historical hallucinations.
  arXiv  Detail & Related papers  (2026-02-27T04:49:01Z)
• Adaptive Detector-Verifier Framework for Zero-Shot Polyp Detection in Open-World Settings [7.768426131383283]
  Existing approaches struggle with the domain gap between controlled laboratory conditions and clinical practice.<n>We propose AdaptiveDetector, a novel two-stage detector-verifier framework comprising a YOLOv11 detector with a vision-language model (VLM) verifier.<n>This combination of adaptive thresholding and cost-sensitive reinforcement learning achieves clinically aligned, open-world polyp detection with substantially fewer false negatives.
  arXiv  Detail & Related papers  (2025-12-13T23:33:05Z)
• Conformal Lesion Segmentation for 3D Medical Images [82.92159832699583]
  We propose a risk-constrained framework that calibrates data-driven thresholds via conformalization to ensure the test-time FNR remains below a target tolerance.<n>We validate the statistical soundness and predictive performance of CLS on six 3D-LS datasets across five backbone models, and conclude with actionable insights for deploying risk-aware segmentation in clinical practice.
  arXiv  Detail & Related papers  (2025-10-19T08:21:00Z)
• Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases [48.87360916431396]
  We introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references.<n>We propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey.<n>Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc.
  arXiv  Detail & Related papers  (2025-03-06T18:35:39Z)
• CLIMB: A Benchmark of Clinical Bias in Large Language Models [39.82307008221118]
  Large language models (LLMs) are increasingly applied to clinical decision-making. Their potential to exhibit bias poses significant risks to clinical equity. Currently, there is a lack of benchmarks that systematically evaluate such clinical bias in LLMs.
  arXiv  Detail & Related papers  (2024-07-07T03:41:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.