What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift

• URL: http://arxiv.org/abs/2608.25251v1
• Date: Wed, 26 Aug 2026 00:19:45 GMT
• Title: What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift
• Abstract summary: Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes.<n>We first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics.<n>We then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information.
• Score: 3.5023163663015286
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical vision-language models (VLMs) can appear reliable in-domain while failing when acquisition domain, paired supervision, or evaluation protocol changes. We study this failure mode as a representation-level blind spot relevant to epistemic intelligence, without claiming a formal estimator of epistemic uncertainty. Using NIH ChestXray14 and CheXpert, we first isolate source-only cross-dataset visual transfer from unsupervised domain-adaptation diagnostics. Using PadChest and OpenI, we then evaluate multimodal alignment under strict pair-index retrieval and quantify metadata-derived source-proxy information retained in frozen embeddings. Self-supervised visual initialization improves NIH-to-CheXpert transfer over supervised ImageNet initialization in matched ResNet-18 comparisons, whereas adversarial adaptation is useful only in a narrow regime and becomes unstable as adversarial pressure increases. Multimodal exact-pair retrieval remains low under external OpenI stress testing, and source-proxy information remains recoverable from learned representations. Qualitative nearest-neighbor and Grad-CAM analyses show clinically plausible cross-dataset structure and thoracic attention patterns in many cases, while device-heavy and false-positive cases remain ambiguous. Auxiliary architecture checks are task-dependent and do not support a universal backbone ranking. Overall, the study shows that apparent competence under a single protocol can conceal transfer, alignment, and shortcut-related failure modes, motivating stress-tested evaluation of medical VLMs under distribution shift.

Related papers

• Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence [45.78409219344429]
  Multimodal large language models (MLLMs) show strong promise for clinical VQA and radiology report generation, yet inference-time hallucinations undermine trustworthy use.<n>We propose a holistic, training-free evidence-injection framework that systematically mitigates hallucinations through dual-side evidence injection.<n>We conduct systematic evaluations on 2 tasks and 5 datasets using textttLLaVA-1.5-7B, texttLLaVA-Med-1.5-7B, texttQwen3-VL-8B/32B, and textttInternVL
  arXiv  Detail & Related papers  (2026-06-30T08:07:25Z)
• Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection [56.362776482614976]
  Within-dataset class-split evaluation is widely used as a proxy for fully unconditional out-of-distribution anomaly detection.<n>We show that this protocol can become ill-posed when the held-out anomaly class overlaps the normal mixture in representation space.<n>We introduce a simple training-free diagnostic, neighborhood class leakage, and show that it predicts score-direction instability across Fashion-MNIST, CIFAR-10, and Imagenette.
  arXiv  Detail & Related papers  (2026-05-23T06:27:42Z)
• Neural Uncertainty Principle: A Unified View of Adversarial Fragility and LLM Hallucination [60.197429875410286]
  Adversarial vulnerability in vision and hallucination in large language models are conventionally viewed as separate problems.<n>Formalizing a Neural Uncertainty Principle (NUP) under a loss-induced state, we find that in near-bound regimes, further compression must be accompanied by increased sensitivity dispersion.<n>In vision, masking highly coupled components improves robustness without costly adversarial training.<n>In language, the same prefill-stage probe detects hallucination risk before generating any answer tokens.
  arXiv  Detail & Related papers  (2026-03-20T02:07:10Z)
• CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models [14.039516906417724]
  We propose a chain-of-distribution framework that constructs clinically plausible pipeline shifts by acquisition-like shading, reconstruction and display remapping, and delivery and export degradations.<n>Across brain MRI, chest X-ray, and abdominal CT, CoDA substantially degrades the zero-shot performance of CLIP-style MVLMs.<n>We also evaluate multimodal large language models (MLLMs) as technical-authenticity auditors of imaging realism and quality rather than pathology.
  arXiv  Detail & Related papers  (2026-03-19T07:00:44Z)
• PF-DAformer: Proximal Femur Segmentation via Domain Adaptive Transformer for Dual-Center QCT [8.358409792893278]
  We develop a domain-adaptive transformer segmentation framework tailored for multi-institutional Quantitative computed tomography (QCT)<n>Our model is trained and validated on one of the largest hip fracture related research cohorts to date, comprising 1,024 QCT images scans from Tulane University and 384 scans from Rochester, Minnesota for proximal femur segmentation.
  arXiv  Detail & Related papers  (2025-10-30T18:07:56Z)
• Learning from Heterogeneous Structural MRI via Collaborative Domain Adaptation for Late-Life Depression Assessment [24.340328016766183]
  We propose a Collaborative Domain Adaptation framework for LLD detection using T1-weighted MRIs.<n>The framework consists of three stages: supervised training on labeled source data, self-supervised target feature adaptation and collaborative training on unlabeled target data.<n>Experiments conducted on multi-site T1-weighted MRI data demonstrate that the framework consistently outperforms state-of-the-art unsupervised domain adaptation methods.
  arXiv  Detail & Related papers  (2025-07-30T01:38:32Z)
• U2AD: Uncertainty-based Unsupervised Anomaly Detection Framework for Detecting T2 Hyperintensity in MRI Spinal Cord [7.811634659561162]
  T2 hyperintensities in spinal cord MR images are crucial biomarkers for conditions such as degenerative cervical myelopathy.<n>Deep learning methods have shown promise in lesion detection, but most supervised approaches are heavily dependent on large, annotated datasets.<n>We propose an Uncertainty-based Unsupervised Anomaly Detection framework, termed U2AD, to address these limitations.
  arXiv  Detail & Related papers  (2025-03-17T17:33:32Z)
• Unsupervised Domain Adaptation for Brain Vessel Segmentation through Transwarp Contrastive Learning [46.248404274124546]
  Unsupervised domain adaptation (UDA) aims to align the labelled source distribution with the unlabelled target distribution to obtain domain-invariant predictive models. This paper proposes a simple yet potent contrastive learning framework for UDA to narrow the inter-domain gap between labelled source and unlabelled target distribution.
  arXiv  Detail & Related papers  (2024-02-23T10:01:22Z)
• Unsupervised Cross-domain Pulmonary Nodule Detection without Source Data [36.61757663123084]
  Cross-domain pulmonary nodule detection suffers from performance degradation due to a large shift of data distributions between the source and target domain. We propose a Source-free Untuning cross-domain method for pulmonary nodule detection (SUP), named Instance-level Contrastive Instruction fine-supervised framework (ICI) We establish a benchmark by adapting a pre-trained source model to three popular datasets for pulmonary nodule detection.
  arXiv  Detail & Related papers  (2023-04-03T15:42:27Z)
• Weakly-Supervised Cross-Domain Adaptation for Endoscopic Lesions Segmentation [79.58311369297635]
  We propose a new weakly-supervised lesions transfer framework, which can explore transferable domain-invariant knowledge across different datasets. A Wasserstein quantified transferability framework is developed to highlight widerange transferable contextual dependencies. A novel self-supervised pseudo label generator is designed to equally provide confident pseudo pixel labels for both hard-to-transfer and easy-to-transfer target samples.
  arXiv  Detail & Related papers  (2020-12-08T02:26:03Z)
• What Can Be Transferred: Unsupervised Domain Adaptation for Endoscopic Lesions Segmentation [51.7837386041158]
  We develop a new unsupervised semantic transfer model including two complementary modules for endoscopic lesions segmentation. Specifically, T_D focuses on where to translate transferable visual information of medical lesions via residual transferability-aware bottleneck. T_F highlights how to augment transferable semantic features of various lesions and automatically ignore untransferable representations.
  arXiv  Detail & Related papers  (2020-04-24T00:57:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.