Look What the Probes Dragged In! Real-World Chest X-ray Shortcuts in MedCLIP

• URL: http://arxiv.org/abs/2608.12086v1
• Date: Wed, 12 Aug 2026 14:08:52 GMT
• Title: Look What the Probes Dragged In! Real-World Chest X-ray Shortcuts in MedCLIP
• Abstract summary: We investigate how real-world shortcuts manifest across different layers of the medical CLIP-based model, MedCLIP.<n>We find that the final linear probes achieve a high AUROC but poor calibration in the models.<n>Patterns consistent with localised shortcuts, such as drains, appear at later layers, while patterns consistent with diffuse shortcuts, such as scanner-specific noise patterns, emerge earlier.
• Score: 1.4336070582473752
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models, such as contrastive language-image pre-training (CLIP)-based approaches, have reached state-of-the-art (SOTA) results in medical artificial intelligence. However, recent work reveals that CLIP-based models remain vulnerable to shortcuts. We investigate how real-world shortcuts manifest across different layers of the medical CLIP-based model, MedCLIP, and its vision encoder, a frozen ResNet-50. We attach 17 linear classification probes to the intermediate layers of the ResNet-50 and train them on three different dataset configurations and targets: NIH-CXR14 (pneumothorax) and PadChest (cardiomegaly and pneumothorax). This setup allows us to observe model behaviour during evaluation using subgroup-based calibration and layer-wise confidence curves. We find that the final linear probes achieve a high AUROC but poor calibration in the models. The layer-wise confidence analyses suggest that shortcuts emerge at different depths. Patterns consistent with localised shortcuts, such as drains, appear at later layers, while patterns consistent with diffuse shortcuts, such as scanner-specific noise patterns, emerge earlier, aligning with previous work. Finally, we conduct a manual analysis of the images, which reveals data quality issues in both NIH-CXR14 and PadChest. Our findings underscore that even SOTA models remain vulnerable to shortcuts, and the need for high-quality and well-annotated datasets to draw solid conclusions. Code can be found on our GitHub: https://github.com/nikodice4/MedCLIP_shortcuts.

Related papers

• SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation [1.5782980044380892]
  Current evaluation protocols for Vision-Language Models in Radiology Report Generation rely on report-level metrics that measure lexical overlap or aggregate clinical correctness.<n>We introduce SHOVIR, a benchmark for evaluating vision shortcut behavior in RRG.
  arXiv  Detail & Related papers  (2026-06-29T12:17:35Z)
• MERGETUNE: Continued fine-tuning of vision-language models [77.8627788911249]
  Fine-tuning vision-language models (VLMs) often leads to catastrophic forgetting of pretrained knowledge.<n>We introduce a novel paradigm, continued fine-tuning (CFT), which seeks to recover pretrained knowledge after a zero-shot model has already been adapted.
  arXiv  Detail & Related papers  (2026-01-15T15:15:53Z)
• Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech [51.14752758616364]
  Speech-based depression detection (SDD) is a promising, non-invasive alternative to traditional clinical assessments.<n>We propose HAREN-CTC, a novel architecture that integrates multi-layer SSL features using cross-attention within a multitask learning framework.<n>The model achieves state-of-the-art macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA, outperforming prior methods across both evaluation scenarios.
  arXiv  Detail & Related papers  (2025-10-05T09:32:12Z)
• Faithful, Interpretable Chest X-ray Diagnosis with Anti-Aliased B-cos Networks [13.303234049048426]
  B-cos networks offer a promising solution by replacing standard linear layers with a weight-input alignment mechanism.<n>Standard B-cos models suffer from severe aliasing artifacts in their explanation maps, making them unsuitable for clinical use.<n>We introduce anti-aliasing strategies using FLC and BlurPool to significantly improve explanation quality.
  arXiv  Detail & Related papers  (2025-07-22T16:56:02Z)
• Label-Efficient Chest X-ray Diagnosis via Partial CLIP Adaptation [0.0]
  This paper proposes a label-efficient strategy for chest X-ray diagnosis.<n>Experiments use the NIH Chest X-ray14 dataset and a pre-trained CLIP ViT-B/32 model.
  arXiv  Detail & Related papers  (2025-07-09T19:57:12Z)
• Interpreting Biomedical VLMs on High-Imbalance Out-of-Distributions: An Insight into BiomedCLIP on Radiology [0.0]
  We analyse the limitations of BiomedCLIP when applied to a highly imbalanced, out-of-distribution medical dataset.<n>We show that the model under zero-shot settings over-predicts all labels, leading to poor precision and inter-class separability.<n>We highlight the need for careful adaptations of the models to foster reliability and applicability in a real-world setting.
  arXiv  Detail & Related papers  (2025-06-17T02:59:42Z)
• Gradient Attention Map Based Verification of Deep Convolutional Neural Networks with Application to X-ray Image Datasets [1.0208529247755187]
  We propose a comprehensive verification framework that evaluates model suitability through multiple complementary strategies.<n>First, we introduce a Gradient Attention Map (GAM)-based approach that analyzes attention patterns using Grad-CAM.<n>Second, we extend verification to early convolutional feature maps, capturing structural mis-alignments missed by attention alone.<n>Third, we incorporate an additional garbage class into the classification model to explicitly reject out-of-distribution inputs.
  arXiv  Detail & Related papers  (2025-04-29T23:41:37Z)
• How Does Pruning Impact Long-Tailed Multi-Label Medical Image Classifiers? [49.35105290167996]
  Pruning has emerged as a powerful technique for compressing deep neural networks, reducing memory usage and inference time without significantly affecting overall performance. This work represents a first step toward understanding the impact of pruning on model behavior in deep long-tailed, multi-label medical image classification.
  arXiv  Detail & Related papers  (2023-08-17T20:40:30Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• Black-Box Diagnosis and Calibration on GAN Intra-Mode Collapse: A Pilot Study [116.05514467222544]
  Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism. One concern raised is whether the state-of-the-art GAN's learned distribution still suffers from mode collapse. This paper explores to diagnose GAN intra-mode collapse and calibrate that, in a novel black-box setting.
  arXiv  Detail & Related papers  (2021-07-23T06:03:55Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.