CLIP Architecture for Abdominal CT Image-Text Alignment and Zero-Shot Learning: Investigating Batch Composition and Data Scaling

• URL: http://arxiv.org/abs/2604.13561v1
• Date: Wed, 15 Apr 2026 07:10:01 GMT
• Title: CLIP Architecture for Abdominal CT Image-Text Alignment and Zero-Shot Learning: Investigating Batch Composition and Data Scaling
• Abstract summary: Vision-language models trained with contrastive learning on paired medical images and reports show strong zero-shot diagnostic capabilities.<n>We reproduce Merlin, a dual-encoder model that aligns 3D abdominal CT volumes with radiology reports using symmetric InfoNCE loss.
• Score: 1.349025992778434
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision-language models trained with contrastive learning on paired medical images and reports show strong zero-shot diagnostic capabilities, yet the effect of training batch composition on learned representations remains unexplored for 3D medical imaging. We reproduce Merlin, a dual-encoder model that aligns 3D abdominal CT volumes with radiology reports using symmetric InfoNCE loss, achieving a zero-shot macro F1 of 74.45% across 30 findings (original: 73.00%). We then investigate two axes of variation. First, we control the normal-to-abnormal ratio within training batches at 25:75, 50:50, and 75:25 using section-level balanced sampling on the full dataset. All three configurations underperform the unbalanced baseline by 2.4 to 2.8 points, with 75:25 achieving the best result (72.02%) among balanced variants. Second, we conduct data scaling ablations on a 4,362-study subset, training with 20%, 40%, and 100% of the data. Performance scales sub-linearly from 65.26% to 71.88%, with individual findings varying dramatically in data sensitivity. Enforcing 50:50 balanced sampling on the same subset further degrades performance to 68.01%, confirming that explicit class balancing hurts regardless of dataset or balancing granularity. Our results indicate that the stochastic diversity of random sampling, combined with Merlin's alternating batching over anatomical subsections, provides more effective regularization than engineered class ratios at the small batch sizes required by 3D medical volumes.

Related papers

• Anatomical Grounding and Leakage-Aware Multimodal Contrastive Learning for Alzheimer's Disease Classification from Structural MRI [0.0]
  Deep networks trained on structural MRI for Alzheimer's disease often reach reasonable accuracy while attending to anatomically irrelevant regions.<n>We use FastSurfer segmentations as an anatomical reference.<n>We adapt a CLIP-style image and organize ADNIMERGE variables along a label-leakage spectrum.
  arXiv  Detail & Related papers  (2026-09-14T17:13:12Z)
• Improving Diagnostic Performance on Small and Imbalanced Datasets Using Class-Based Input Image Composition [0.2676349883103403]
  Small, imbalanced datasets and poor input image quality can lead to high false predictions rates with deep learning models.<n>This paper introduces Class-Based Image Composition, an approach that allows us to reformulate training inputs through a fusion of multiple images of the same class into combined visual composites.
  arXiv  Detail & Related papers  (2025-11-05T22:34:06Z)
• Taylor-Series Expanded Kolmogorov-Arnold Network for Medical Imaging Classification [0.0]
  This study introduces Kolmogorov-Arnold Networks (KANs) for accurate medical image classification with limited, diverse datasets.<n>The models include SBTAYLOR-KAN, integrating B-splines with Taylor series; SBRBF-KAN, combining B-splines with Radial Basis Functions; and SBWAVELET-KAN, embedding B-splines in Morlet wavelet transforms.<n>The models were evaluated on brain MRI, chest X-rays, tuberculosis X-rays, and skin lesion images without preprocessing.
  arXiv  Detail & Related papers  (2025-09-17T04:33:54Z)
• Automated Radiographic Total Sharp Score (ARTSS) in Rheumatoid Arthritis: A Solution to Reduce Inter-Intra Reader Variation and Enhancing Clinical Practice [3.8516555293145345]
  This study introduces an Automated Radiographic Sharp Scoring framework that leverages deep learning to analyze full-hand X-ray images.<n>We developed ARTSS using data from 970 patients, structured into four stages: I) Image pre-processing and re-orientation using ResNet50, II) Hand segmentation using UNet.3, III) Joint identification using YOLOv7, and IV) TSS prediction using models such as VGG16, VGG19, ResNet50, DenseNet201, EfficientNetB0, and Vision Transformer (ViT)
  arXiv  Detail & Related papers  (2025-09-08T16:21:45Z)
• Brain Tumor Classification on MRI in Light of Molecular Markers [56.99710477905796]
  Co-deletion of the 1p/19q gene is associated with clinical outcomes in low-grade gliomas.<n>This study aims to utilize a specially MRI-based convolutional neural network for brain cancer detection.
  arXiv  Detail & Related papers  (2024-09-29T07:04:26Z)
• CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? [72.19502317793133]
  We study the effectiveness of data-balancing for mitigating biases in contrastive language-image pretraining (CLIP) We present a novel algorithm, called Multi-Modal Moment Matching (M4), designed to reduce both representation and association biases.
  arXiv  Detail & Related papers  (2024-03-07T14:43:17Z)
• Few-shot learning for COVID-19 Chest X-Ray Classification with Imbalanced Data: An Inter vs. Intra Domain Study [49.5374512525016]
  Medical image datasets are essential for training models used in computer-aided diagnosis, treatment planning, and medical research. Some challenges are associated with these datasets, including variability in data distribution, data scarcity, and transfer learning issues when using models pre-trained from generic images. We propose a methodology based on Siamese neural networks in which a series of techniques are integrated to mitigate the effects of data scarcity and distribution imbalance.
  arXiv  Detail & Related papers  (2024-01-18T16:59:27Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Active Learning in Brain Tumor Segmentation with Uncertainty Sampling, Annotation Redundancy Restriction, and Data Initialization [17.3513750927719]
  Deep learning models have demonstrated great potential in medical 3D imaging, but their development is limited by the expensive, large volume of annotated data required. Active learning (AL) addresses this by training a model on a subset of the most informative data samples without compromising performance. We compared different AL strategies and propose a framework that minimizes the amount of data needed for state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-02-05T04:45:08Z)
• Density-Aware Personalized Training for Risk Prediction in Imbalanced Medical Data [89.79617468457393]
  Training models with imbalance rate (class density discrepancy) may lead to suboptimal prediction. We propose a framework for training models for this imbalance issue. We demonstrate our model's improved performance in real-world medical datasets.
  arXiv  Detail & Related papers  (2022-07-23T00:39:53Z)
• A self-supervised learning strategy for postoperative brain cavity segmentation simulating resections [46.414990784180546]
  Convolutional neural networks (CNNs) are the state-of-the-art image segmentation technique. CNNs require large annotated datasets for training. Self-supervised learning strategies can leverage unlabeled data for training.
  arXiv  Detail & Related papers  (2021-05-24T12:27:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.