A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

• URL: http://arxiv.org/abs/2606.10066v1
• Date: Mon, 08 Jun 2026 18:40:10 GMT
• Title: A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks
• Abstract summary: Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years.<n>We audit open VLMs on SLAKE-En, PathVQA, VQA-RAD, and an auxiliary public OmniMedVQA mirror using four detector families.
• Score: 46.43161388634735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years, yet reported accuracy assumes these examples were absent from pretraining. We audit open VLMs on SLAKE-En, PathVQA, VQA-RAD, and an auxiliary public OmniMedVQA mirror using four detector families: image-side near-neighbour overlap against PMC-OA-beta, canonical-order exchangeability, cohort-relative Min-K%++ tail enrichment, and cross-model top-K overlap. We find measurable image-side source overlap on SLAKE-En: 19.8% of images are flagged under SigLIP-B-16 and 4.2% under SigLIP-SO400M, while out-of-domain controls produce 0/2000 flags. Manual adjudication shows same-modality, same-projection matches to different patients rather than verified pixel-level duplicates, so we interpret this as source or distributional overlap rather than confirmed per-image memorization. On the text side, Qwen2.5-VL on SLAKE-En shows a canonical-order exchangeability signal that survives ordering ablation and external non-medical baselines. On the OmniMedVQA mirror, exchangeability fires for five medical and general VLMs while BLIP-2 remains clean. In contrast, cohort-relative Min-K%++ tail enrichment and cross-model top-K overlap collapse under an external pre-domain baseline: BLIP-2 reproduces the apparent positive signals despite lacking plausible medical-VQA exposure. We conclude that these cohort-relative detectors are unreliable as standalone membership-inference signals on small medical-VLM cohorts.

Related papers

• Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images [48.646448077284276]
  We propose Spatial-FAD, a framework for effective few-shot medical anomaly detection.<n>It improves lesion localization by combining VFM spatial priors with CLIP semantics.<n>Our method achieves an average improvement of over 11.4% in Dice score and 1.8% in AUC.
  arXiv  Detail & Related papers  (2026-09-11T05:21:01Z)
• What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift [3.5023163663015286]
  Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes.<n>We first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics.<n>We then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information.
  arXiv  Detail & Related papers  (2026-08-26T00:19:45Z)
• Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs [6.775973739193119]
  Large vision language models (VLMs) report strong accuracy on medical question-answering, yet it remains unclear whether they reason from visual evidence or exploit textual shortcuts.<n>We introduce a counterfactual evaluation framework that decouples visual and textual contributions by substituting input images with controlled surrogates blank, pixel-shuffled, image-absent, and CLIP-retrieved hard negatives.
  arXiv  Detail & Related papers  (2026-07-04T00:06:35Z)
• Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain [43.675927916974466]
  We introduce the Spectral Alignment Score (SAS), an asymmetric metric that projects both modalities onto the principal eigenbasis of an anchor modality and computes eigenvalue-weighted per-eigenmode correlations, resulting in directional scores whose difference quantifies modality information imbalance.<n>Our experiments show that medical images retain richer structural information than their paired clinical reports, a directional asymmetry invisible to all competing metrics, and that SAS achieves the strongest zero-label correlation with retrieval performance in the medical domain, positioning it as a practical diagnostic tool for clinical deployment.
  arXiv  Detail & Related papers  (2026-06-03T08:50:30Z)
• Cross-modal linkage risk in clinical vision-language models [1.828619151598757]
  Vision-language models (VLMs) trained on paired chest radiographs and radiology reports learn a shared embedding space that can preserve instance-level image-report correspondence.<n>This poses a privacy risk in settings where radiographs and reports are deliberately kept separate after acquisition.<n>We formalized this as image-to-report retrieval and used public paired cohorts to audit the risk rather than as the privacy scenario.
  arXiv  Detail & Related papers  (2026-06-01T14:01:46Z)
• LiteMedCoT-VL: Parameter-Efficient Adaptation for Medical Visual Question Answering [1.0559587981237513]
  We introduce LiteMedCoT-VL, a pipeline that transfers chain-of-thought reasoning from a 235B teacher model to 2B student models.<n>On the PMC-VQA benchmark, LiteMedCoT-VL achieves 64.9% accuracy, exceeding the zero-shot Qwen3-VL-4B baseline of 53.9% by 11.0 percentage points.
  arXiv  Detail & Related papers  (2026-05-10T07:21:33Z)
• Auditing Frontier Vision-Language Models for Trustworthy Medical VQA: Grounding Failures, Format Collapse, and Domain Adaptation [4.177028541321909]
  We audit five vision-language models (VLMs) on Medical VQA along two trust-relevant axes.<n>Models localize anatomical and pathological targets poorly.<n>A self-grounding pipeline, where the same model localizes then answers, degrades VQA accuracy.
  arXiv  Detail & Related papers  (2026-04-30T11:11:47Z)
• Agri-CPJ: A Training-Free Explainable Framework for Agricultural Pest Diagnosis Using Caption-Prompt-Judge and LLM-as-a-Judge [40.23111543895404]
  This paper describes Agri-CPJ (Caption-Prompt-Judge), a training-free few-shot framework in which a large vision-language model first generates a structured morphological caption.<n>Two candidate responses are then generated from complementary viewpoints, and an LLM judge selects the stronger one based on domain-specific criteria.<n>On CDDMBench, pairing GPT-5-Nano with GPT-5-mini-generated captions yields textbf+22.7 pp in disease classification and textbf+19.5 points in QA score over no-caption baselines.
  arXiv  Detail & Related papers  (2026-04-26T13:33:45Z)
• Maximizing T2-Only Prostate Cancer Localization from Expected Diffusion Weighted Imaging [68.57480103120561]
  Multiparametric MRI is increasingly recommended as a first-line noninvasive approach to detect and localize prostate cancer.<n>Early machine learning attempts using only T2w images have shown promising diagnostic performance in segmenting radiologist-annotated lesions.<n>This work investigates an arguably more challenging application using only T2w at inference, but to localize individual cancers based on independent histopathology labels.
  arXiv  Detail & Related papers  (2026-04-01T14:50:18Z)
• Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models [2.064612766965483]
  We show a model can achieve perfect consistency by relying on text patterns rather than the input image.<n>We introduce a four-quadrant per-sample safety taxonomy that jointly evaluates consistency (stable predictions across paraphrased prompts) and image reliance.
  arXiv  Detail & Related papers  (2026-03-22T00:06:53Z)
• MeCaMIL: Causality-Aware Multiple Instance Learning for Fair and Interpretable Whole Slide Image Diagnosis [40.3028468133626]
  Multiple instance learning (MIL) has emerged as the dominant paradigm for whole slide image (WSI) analysis in computational pathology.<n>textbfMeCaMIL, a causality-aware MIL framework, explicitly models demographic confounders through structured causal graphs.<n>MeCaMIL achieves superior fairness -- demographic disparity variance drops by over 65% relative reduction on average across attributes.
  arXiv  Detail & Related papers  (2025-11-14T06:47:21Z)
• Intra-video Positive Pairs in Self-Supervised Learning for Ultrasound [65.23740556896654]
  Self-supervised learning (SSL) is one strategy for addressing the paucity of labelled data in medical imaging. In this study, we investigated the effect of utilizing proximal, distinct images from the same B-mode ultrasound video as pairs for SSL. Named Intra-Video Positive Pairs (IVPP), the method surpassed previous ultrasound-specific contrastive learning methods' average test accuracy on COVID-19 classification.
  arXiv  Detail & Related papers  (2024-03-12T14:57:57Z)
• Task-Based Assessment for Neural Networks: Evaluating Undersampled MRI Reconstructions based on Human Observer Signal Detection [45.82374977939355]
  Common metrics for evaluating image quality like the normalized root mean squared error (NRMSE) and structural similarity (SSIM) are global metrics which average out impact of subtle features in the images. We used measures of image quality which incorporate a subtle signal for a specific task allow for image quality assessment which locally evaluates the effect of undersampling on a signal.
  arXiv  Detail & Related papers  (2022-10-21T16:39:04Z)
• Transformers Improve Breast Cancer Diagnosis from Unregistered Multi-View Mammograms [6.084894198369222]
  We leverage the architecture of Multi-view Vision Transformers to capture long-range relationships of multiple mammograms from the same patient in one examination. Our four-image (two-view-two-side) Transformer-based model achieves case classification with an area under ROC curve (AUC = 0.818) It also outperforms two one-view-two-side models that achieve AUC of 0.724 (CC view) and 0.769 (MLO view)
  arXiv  Detail & Related papers  (2022-06-21T03:54:21Z)
• Improved Slice-wise Tumour Detection in Brain MRIs by Computing Dissimilarities between Latent Representations [68.8204255655161]
  Anomaly detection for Magnetic Resonance Images (MRIs) can be solved with unsupervised methods. We have proposed a slice-wise semi-supervised method for tumour detection based on the computation of a dissimilarity function in the latent space of a Variational AutoEncoder. We show that by training the models on higher resolution images and by improving the quality of the reconstructions, we obtain results which are comparable with different baselines.
  arXiv  Detail & Related papers  (2020-07-24T14:02:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.