CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs

• URL: http://arxiv.org/abs/2605.05810v1
• Date: Thu, 07 May 2026 07:46:17 GMT
• Title: CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs
• Abstract summary: CXR-ContraBench is a diagnostic benchmark spanning internal ReXVQA slices and external OpenI and CheXpert protocols.<n>We study this failure as negated-option attraction, where a model is drawn to a negated answer option even when it conflicts with both the visual evidence and the question.
• Score: 20.96410413299322
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When a chest X-ray shows consolidation but the question asks which finding is present, a medical vision-language model may answer "No consolidation." This is more than an incorrect choice: it is a polarity reversal that emits a clinical statement contradicting the image. We study this failure as negated-option attraction, where a model is drawn to a negated answer option even when it conflicts with both the visual evidence and the question. We introduce CXR-ContraBench (Chest X-Ray Contradiction Benchmark), a diagnostic benchmark spanning internal ReXVQA slices and external OpenI and CheXpert protocols. The benchmark centers on present-finding questions, where selecting "No X" despite visible X creates the main clinical risk, and uses absent-finding questions as secondary tests of whether models copy negated wording. Across CheXpert protocols, the failure is substantial and persistent. On a strict direct presence probe, MedGemma and Qwen2.5-VL reach only 31.49% and 30.21% accuracy, respectively; on a matched 135,754-record CheXpert training-split protocol, both models select negated options on over 62% of presence questions. Chain-of-thought prompting reduces some presence-side reversals but does not eliminate them and can amplify absence-side contradictions. Finally, QCCV-Neg (Question-Conditioned Consistency Verifier for Negation) deterministically repairs the measured polarity-confused subset without retraining, raising MedGemma and Qwen2.5-VL to 96.60% and 95.32% accuracy on the direct presence probe. These results show that standard accuracy can hide a clinically meaningful inference-time polarity failure. Source code and benchmark construction scripts are available at https://github.com/fangzr/cxr-contrabench-code.

Related papers

• MedQA-MM: Shortcuts Behind Medical Visual Reasoning [45.333881843709804]
  A benchmark score credits final answers, but not the route by which an item can be answered.<n>In medical multimodal multiple-choice questions (MCQs), this distinction matters because a correct answer can be supported by the intended image finding.<n>Here, a route is an observable input path that can support answer selection, not a claim about the model's hidden cognition.
  arXiv  Detail & Related papers  (2026-09-03T01:40:19Z)
• MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models [2.2090506971647144]
  Vision-language models (VLMs) are increasingly used in clinical pipelines where a chest X-ray is interpreted alongside retrieved reports, preliminary notes, or prior imaging.<n>Existing benchmarks measure whether models answer correctly in isolation, but not whether they preserve a correct image-only decision when plausible context conflicts with the image.<n>We introduce Multi-Context Chest X-ray (MC-CXR), a benchmark of 240 cases expanded into 2,522 instances that isolates context-induced disruption through paired perturbation.
  arXiv  Detail & Related papers  (2026-08-25T06:28:17Z)
• Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMs [2.064612766965483]
  Attention and saliency heatmaps are widely used to explain medical Vision-Language Model (VLM) outputs on chest X-rays.<n>We audit faithfulness via overlap with radiologist bounding boxes on PadChest.<n>We study three MedGemma-4B variants, cross-family probes on LLaVA-RAD and Qwen3-VL-8B-Instruct, and the specialist CheXagent-2-3b.
  arXiv  Detail & Related papers  (2026-07-20T23:19:28Z)
• Confidence is Not Reliability: Rethinking MC Dropout in Brain Tumour Segmentation [2.104759809264928]
  A segmentation model that fails silently on treatment-critical sub-regions represents a patient safety risk.<n>We ask whether voxel-level uncertainty estimation via Monte Carlo (MC) Dropout can reliably identify segmentation errors.<n>We evaluate a high-performance pretrained SegResNet and a locally trained UNet with residual units.
  arXiv  Detail & Related papers  (2026-06-17T17:24:41Z)
• A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks [46.43161388634735]
  Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years.<n>We audit open VLMs on SLAKE-En, PathVQA, VQA-RAD, and an auxiliary public OmniMedVQA mirror using four detector families.
  arXiv  Detail & Related papers  (2026-06-08T18:40:10Z)
• Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection [56.362776482614976]
  Within-dataset class-split evaluation is widely used as a proxy for fully unconditional out-of-distribution anomaly detection.<n>We show that this protocol can become ill-posed when the held-out anomaly class overlaps the normal mixture in representation space.<n>We introduce a simple training-free diagnostic, neighborhood class leakage, and show that it predicts score-direction instability across Fashion-MNIST, CIFAR-10, and Imagenette.
  arXiv  Detail & Related papers  (2026-05-23T06:27:42Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretation [48.23500302185585]
  We present CheXOne, a reasoning-enabled vision-language model for CXR interpretation.<n>CheXOne jointly generates diagnostic predictions and explicit, clinically grounded reasoning traces.<n>We show that CheXOne-drafted reports are comparable to or better than resident-written reports in 55% of cases.
  arXiv  Detail & Related papers  (2026-04-01T05:19:09Z)
• AdURA-Net: Adaptive Uncertainty and Region-Aware Network [0.7771558179849474]
  In clinical decision-making, the uncertain label plays a tricky role as the model should not be forced to provide a confident prediction.<n>Here, we propose AdURA-Net, a geometry-driven adaptive uncertainty-aware framework for reliable thoracic disease classification.
  arXiv  Detail & Related papers  (2026-02-27T08:56:24Z)
• Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering [94.37535002230504]
  We develop a training-free, inference-time control framework termed Semantically Decoupled Latent Steering.<n>Our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition.<n>We show that our approach significantly reduces the probability of historical hallucinations.
  arXiv  Detail & Related papers  (2026-02-27T04:49:01Z)
• Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learning [10.264467364282865]
  Similarity-as-Evidence (SaE) calibrates text-image similarities by introducing a Similarity Evidence Head (SEH)<n>SaE attains state-of-the-art macro-averaged accuracy of 82.57% on medical imaging datasets with a 20% label budget.
  arXiv  Detail & Related papers  (2026-02-21T15:21:54Z)
• Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations [19.488236277427358]
  Vision-language models (VLMs) often produce chain-of-thought (CoT) explanations that sound plausible yet fail to reflect the underlying decision process.<n>We present a clinically grounded framework for chest X-ray visual question answering (VQA) that probes CoT faithfulness via controlled text and image modifications.
  arXiv  Detail & Related papers  (2025-10-13T09:28:22Z)
• Towards Reliable Medical Image Segmentation by Modeling Evidential Calibrated Uncertainty [57.023423137202485]
  Concerns regarding the reliability of medical image segmentation persist among clinicians.<n>We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks.<n>By leveraging subjective logic theory, we explicitly model probability and uncertainty for medical image segmentation.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)
• Interpreting Chest X-rays via CNNs that Exploit Hierarchical Disease Dependencies and Uncertainty Labels [0.33598755777055367]
  We present a framework based on deep convolutional neural networks (CNNs) for diagnos-ing the presence of 14 common thoracic diseases and observations. The proposed method was also evaluated on an inde-pendent test set of the CheXpert competition, containing 500 CXR studies annotated by apanel of 5 experienced radiologists.
  arXiv  Detail & Related papers  (2020-05-25T11:07:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.