Cross-modal linkage risk in clinical vision-language models

• URL: http://arxiv.org/abs/2606.02276v1
• Date: Mon, 01 Jun 2026 14:01:46 GMT
• Title: Cross-modal linkage risk in clinical vision-language models
• Abstract summary: Vision-language models (VLMs) trained on paired chest radiographs and radiology reports learn a shared embedding space that can preserve instance-level image-report correspondence.<n>This poses a privacy risk in settings where radiographs and reports are deliberately kept separate after acquisition.<n>We formalized this as image-to-report retrieval and used public paired cohorts to audit the risk rather than as the privacy scenario.
• Score: 1.828619151598757
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) trained on paired chest radiographs and radiology reports learn a shared embedding space that can preserve instance-level image-report correspondence. This poses a privacy risk in settings where radiographs and reports are deliberately kept separate after acquisition, such as image-only data sharing or access-controlled reports, because a de-identified image may be re-linked to its original narrative report through cosine similarity alone. We formalized this as image-to-report retrieval and used public paired cohorts, in which the true pairing is known by design, as ground-truth benchmarks to audit the risk rather than as the privacy scenario. Evaluating VLMs of increasing clinical specialization on 406,241 paired examples from 126,804 patients across MIMIC-CXR (43,793 held-out pairs) and external CheXpert Plus (29,296 pairs), we found that re-linkage rose systematically with specialization: the strongest VLM retrieved the correct report at 15 times chance at a candidate pool of N = 100, 50 times chance at N = 10,000, and well above chance at full-database scale. The signal persisted under pathology-matched hard negatives that removed disease-label shortcuts, indicating correspondence beyond broad diagnostic categories. To reduce it without retraining, we froze both encoders and applied differentially private optimization only to the projection heads defining the alignment layer (epsilon = 0.34, delta = 6x10-6). This reduced Recall@1 by 61.8% at N = 10,000 on MIMIC-CXR and transferred to CheXpert Plus without retraining, while image-side utility was largely preserved: macro AUROC for linear-probe classification across 14 labels shifted only from 79.63% to 79.43%. Targeted DP finetuning of the shared alignment layer can substantially reduce cross-modal re-linkage without materially degrading the image representations that make these models clinically useful.

Related papers

• Anatomical Grounding and Leakage-Aware Multimodal Contrastive Learning for Alzheimer's Disease Classification from Structural MRI [0.0]
  Deep networks trained on structural MRI for Alzheimer's disease often reach reasonable accuracy while attending to anatomically irrelevant regions.<n>We use FastSurfer segmentations as an anatomical reference.<n>We adapt a CLIP-style image and organize ADNIMERGE variables along a label-leakage spectrum.
  arXiv  Detail & Related papers  (2026-09-14T17:13:12Z)
• Text-Guided Diffusion-Based Adversarial Attacks on Chest X-Ray Images [1.167393095135318]
  We propose a text-guided diffusion-based adversarial framework that optimize learnable text conditioning while keeping the diffusion generator and target classifier frozen.<n>Our approach consistently produced the greatest degradation in classifier performance, reducing AUROC to 0.3885-0.5646 in binary classification and 0.4441-0.4878 in the multi-disease setting.<n>These findings reveal a clinically important discrepancy between human and machine interpretation and demonstrate the need to extend medical AI robustness evaluation beyond conventional pixel-space attacks.
  arXiv  Detail & Related papers  (2026-08-29T22:19:04Z)
• MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models [2.2090506971647144]
  Vision-language models (VLMs) are increasingly used in clinical pipelines where a chest X-ray is interpreted alongside retrieved reports, preliminary notes, or prior imaging.<n>Existing benchmarks measure whether models answer correctly in isolation, but not whether they preserve a correct image-only decision when plausible context conflicts with the image.<n>We introduce Multi-Context Chest X-ray (MC-CXR), a benchmark of 240 cases expanded into 2,522 instances that isolates context-induced disruption through paired perturbation.
  arXiv  Detail & Related papers  (2026-08-25T06:28:17Z)
• A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports [58.14965296904734]
  EndoCLIP is a vision-language foundation model trained on 125,756 lesion-level image-text pairs.<n>It outperforms general-purpose and biomedical vision-language encoders in both zero-shot and linear-probe settings.
  arXiv  Detail & Related papers  (2026-07-30T16:24:36Z)
• A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks [46.43161388634735]
  Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years.<n>We audit open VLMs on SLAKE-En, PathVQA, VQA-RAD, and an auxiliary public OmniMedVQA mirror using four detector families.
  arXiv  Detail & Related papers  (2026-06-08T18:40:10Z)
• A Vision-language Framework for Comparative Reasoning in Radiology [68.52471827773482]
  We formulate radiological comparison as an entity-aware cross-image reasoning problem.<n>We construct a large-scale comparative imaging resource derived from routine image-report pairs.<n>We develop MedReCo, an entity-aware visual encoder for controllable retrieval of clinically analogous cases.
  arXiv  Detail & Related papers  (2026-06-04T17:12:47Z)
• Magnification-Invariant Image Classification via Domain Generalization and Stable Sparse Embedding Signatures [0.0]
  Magnification shift is a major obstacle to robust histopathology classification, because models trained on one scale often generalize poorly to another.<n>We evaluated this problem on the BreaKHis dataset using a strict patient-disjoint leave-one-magnification-out protocol.
  arXiv  Detail & Related papers  (2026-04-28T16:26:14Z)
• Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering [94.37535002230504]
  We develop a training-free, inference-time control framework termed Semantically Decoupled Latent Steering.<n>Our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition.<n>We show that our approach significantly reduces the probability of historical hallucinations.
  arXiv  Detail & Related papers  (2026-02-27T04:49:01Z)
• One-shot synthesis of rare gastrointestinal lesions improves diagnostic accuracy and clinical training [45.49415063761575]
  EndoRare is a one-shot, retraining-free generative framework that synthesizes diverse, high-fidelity lesion exemplars from a single reference image.<n>We validated the framework across four rare pathologies.<n>These results establish a practical, data-efficient pathway to bridge the rare-disease gap in both computer-aided diagnostics and clinical education.
  arXiv  Detail & Related papers  (2025-12-30T15:07:09Z)
• I Detect What I Don't Know: Incremental Anomaly Learning with Stochastic Weight Averaging-Gaussian for Oracle-Free Medical Imaging [2.384534878752428]
  We introduce an unsupervised, oracle-free framework that incrementally expands a trusted set of normal samples without any anomaly labels.<n>A frozen pretrained vision backbone is augmented with tiny convolutional adapters, ensuring rapid domain adaptation with negligible computational overhead.<n>On COVID-CXR, ROC-AUC improves from 0.9489 to 0.9982; on Pneumonia CXR, ROC-AUC rises from 0.6834 to 0.8968; and on Brain MRI ND-5, ROC-AUC increases from 0.6041 to 0.7269.
  arXiv  Detail & Related papers  (2025-11-05T23:28:14Z)
• Revisiting Computer-Aided Tuberculosis Diagnosis [56.80999479735375]
  Tuberculosis (TB) is a major global health threat, causing millions of deaths annually. Computer-aided tuberculosis diagnosis (CTD) using deep learning has shown promise, but progress is hindered by limited training data. We establish a large-scale dataset, namely the Tuberculosis X-ray (TBX11K) dataset, which contains 11,200 chest X-ray (CXR) images with corresponding bounding box annotations for TB areas. This dataset enables the training of sophisticated detectors for high-quality CTD.
  arXiv  Detail & Related papers  (2023-07-06T08:27:48Z)
• Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders [50.689585476660554]
  We propose a new fine-tuning strategy that includes positive-pair loss relaxation and random sentence sampling. Our approach consistently improves overall zero-shot pathology classification across four chest X-ray datasets and three pre-trained models.
  arXiv  Detail & Related papers  (2022-12-14T06:04:18Z)
• Generative Adversarial Networks for Weakly Supervised Generation and Evaluation of Brain Tumor Segmentations on MR Images [0.0]
  This work presents a weakly supervised approach to segment anomalies in 2D magnetic resonance images. We train a generative adversarial network (GAN) that converts cancerous images to healthy variants. Non-cancerous variants can also be used to evaluate the segmentations in a weakly supervised fashion.
  arXiv  Detail & Related papers  (2022-11-10T00:04:46Z)
• Feature-enhanced Adversarial Semi-supervised Semantic Segmentation Network for Pulmonary Embolism Annotation [6.142272540492936]
  This study established a feature-enhanced adversarial semi-supervised semantic segmentation model to automatically annotate pulmonary embolism lesion areas. In current studies, all of the PEA image segmentation methods are trained by supervised learning. This study proposed a semi-supervised learning method to make the model applicable to different datasets by adding a small amount of unlabeled images.
  arXiv  Detail & Related papers  (2022-04-08T04:21:02Z)
• Improved Slice-wise Tumour Detection in Brain MRIs by Computing Dissimilarities between Latent Representations [68.8204255655161]
  Anomaly detection for Magnetic Resonance Images (MRIs) can be solved with unsupervised methods. We have proposed a slice-wise semi-supervised method for tumour detection based on the computation of a dissimilarity function in the latent space of a Variational AutoEncoder. We show that by training the models on higher resolution images and by improving the quality of the reconstructions, we obtain results which are comparable with different baselines.
  arXiv  Detail & Related papers  (2020-07-24T14:02:09Z)
• Co-Heterogeneous and Adaptive Segmentation from Multi-Source and Multi-Phase CT Imaging Data: A Study on Pathological Liver and Lesion Segmentation [48.504790189796836]
  We present a novel segmentation strategy, co-heterogenous and adaptive segmentation (CHASe) We propose a versatile framework that fuses appearance based semi-supervision, mask based adversarial domain adaptation, and pseudo-labeling. CHASe can further improve pathological liver mask Dice-Sorensen coefficients by ranges of $4.2% sim 9.4%$.
  arXiv  Detail & Related papers  (2020-05-27T06:58:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.