Representation geometry shapes task performance in vision-language modeling for CT enterography

• URL: http://arxiv.org/abs/2604.13021v1
• Date: Tue, 14 Apr 2026 17:56:23 GMT
• Title: Representation geometry shapes task performance in vision-language modeling for CT enterography
• Abstract summary: We present the first study of vision-language transfer learning on abdominal CT enterography.<n> mean pooling of slice embeddings gives better categorical disease assessment.<n>Per-slice tissue contrast matters more than broader spatial coverage.
• Score: 1.1999555634662633
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computed tomography (CT) enterography is a primary imaging modality for assessing inflammatory bowel disease (IBD), yet the representational choices that best support automated analysis of this modality are unknown. We present the first study of vision-language transfer learning on abdominal CT enterography and identify two main findings. First, mean pooling of slice embeddings gives better categorical disease assessment (59.2\% three-class accuracy), whereas attention pooling gives better cross-modal retrieval (0.235 text-to-image MRR). This pattern holds across all LoRA configurations tested and suggests that the two aggregators emphasize different properties of the learned representation. Second, per-slice tissue contrast matters more than broader spatial coverage: multi-window RGB encoding, which maps complementary Hounsfield Unit windows to RGB channels, outperforms all strategies that increase spatial coverage through multiplanar sampling, and in this setting adding coronal and sagittal views reduces classification performance. For report generation, fine-tuning without retrieval context yields within-1 severity accuracy at the prevalence-matched chance level (70.4\% vs.\ 71\% random), suggesting little learned ordering beyond the class distribution. Retrieval-augmented generation (RAG) improves this across all configurations, scoring 7--14 percentage points above the chance baseline and improving ordinal MAE from 0.98 to 0.80--0.89. A three-teacher pseudolabel framework enables all comparisons without expert annotations. Together, these findings provide the first baselines for this underexplored modality and offer practical guidance for building vision-language systems for volumetric medical imaging.

Related papers

• Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images [48.646448077284276]
  We propose Spatial-FAD, a framework for effective few-shot medical anomaly detection.<n>It improves lesion localization by combining VFM spatial priors with CLIP semantics.<n>Our method achieves an average improvement of over 11.4% in Dice score and 1.8% in AUC.
  arXiv  Detail & Related papers  (2026-09-11T05:21:01Z)
• Bag-of-Visual-Words for Spatial Mapping of Lung Adenocarcinoma Growth Patterns [2.3230247783144873]
  Bag-of-Visual-Words (BoVW) pipeline learns a visual vocabulary from frozen foundation model embeddings extracted from annotated regions of interest (ROIs)<n>Pattern prototypes are constructed as mean BoVW histograms of same-label ROIs and used for nearest-prototype classification of sliding-window regions under Jensen--Shannon divergence.<n>We evaluate the method on 87 CPTAC-LUAD patients using three foundation model encoders and multiple vocabulary sizes on two clinically motivated tasks.
  arXiv  Detail & Related papers  (2026-08-05T17:17:36Z)
• PulmoSight-XAI: An Explainable Multi-View Attention Ensemble with Gradient Boosting Meta-Learning for Multi-Label Chest X-Ray Classification [1.2744523252873352]
  We propose an explainable hierarchical multi-view ensemble framework for the robust classification of 14 thoracic pathologies.<n>The framework employs view-specific training by independently modeling frontal and lateral radiographs.<n>The framework achieves state-of-the-art macro-average AUROC scores of 0.9319 for frontal and 0.9154 for lateral radiographs.
  arXiv  Detail & Related papers  (2026-07-05T19:50:53Z)
• Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Models [40.432049062995326]
  Current prompt-based and adapter-based tuning of vision-language models (VLMs) is attractive for medical imaging.<n>We propose Omni-Geometry Knowledge Distillation (OGKD), a new framework that injects class-relation structure into the teacher to produce directional targets.<n>Our OGKD achieves substantially better performance, consistently improving accuracy by an average absolute gain of 1.7%-2.8% over all prior state-of-the-art VLM adaptation counterparts.
  arXiv  Detail & Related papers  (2026-06-03T14:17:57Z)
• Comparison of ConvNeXt and Vision-Language Models for Breast Density Assessment in Screening Mammography [39.58317527488534]
  This study compares multimodal and CNN-based methods for automated classification using the BI-RADS system.<n>Zero-shot classification achieved modest performance, while the fine-tuned ConvNeXt model outperformed the BioMedCLIP linear probe.<n>These findings suggest that despite the promise of multimodal learning, CNN-based models with end-to-end fine-tuning provide stronger performance for specialized medical imaging.
  arXiv  Detail & Related papers  (2025-06-16T20:14:37Z)
• MedFocusCLIP : Improving few shot classification in medical datasets using pixel wise attention [1.2277343096128712]
  We propose to leverage advanced segmentation capabilities of Segment Anything Model 2 (SAM2) as a visual prompting cue to help visual encoder in the CLIP (Contrastive Language-Image Pretraining)<n>This helps the model to focus on highly discriminative regions, without getting distracted from visually similar background features.<n>We evaluate our method on diverse medical datasets including X-rays, CT scans, and MRI images, and report an accuracy of (71%, 81%, 86%, 58%) from the proposed approach.
  arXiv  Detail & Related papers  (2025-01-07T14:49:12Z)
• Analysis of the Two-Step Heterogeneous Transfer Learning for Laryngeal Blood Vessel Classification: Issue and Improvement [8.7937485450551]
  This work pioneers exploring two-step heterogeneous transfer learning (THTL) for laryngeal lesion classification. We use diabetic retinopathy color fundus images, semantically non-identical yet vascular images, as the intermediate domain. We introduce an enhanced fine-tuning strategy in THTL called Step-Wise Fine-Tuning (SWFT) and apply it to the ResNet models.
  arXiv  Detail & Related papers  (2024-02-29T09:52:39Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders [50.689585476660554]
  We propose a new fine-tuning strategy that includes positive-pair loss relaxation and random sentence sampling. Our approach consistently improves overall zero-shot pathology classification across four chest X-ray datasets and three pre-trained models.
  arXiv  Detail & Related papers  (2022-12-14T06:04:18Z)
• Assessing Coarse-to-Fine Deep Learning Models for Optic Disc and Cup Segmentation in Fundus Images [0.0]
  coarse-to-fine deep learning algorithms are used to efficiently measure the vertical cup-to-disc ratio (vCDR) in fundus images. We present a comprehensive analysis of different coarse-to-fine designs for OD/OC segmentation using 5 public databases. Our analysis shows that these algorithms not necessarily outperfom standard multi-class single-stage models.
  arXiv  Detail & Related papers  (2022-09-28T19:19:16Z)
• HistoPerm: A Permutation-Based View Generation Approach for Improving Histopathologic Feature Representation Learning [33.1098457952173]
  HistoPerm is a view generation method for representation learning using joint embedding architectures. HistoPerm permutes augmented views of patches extracted from whole-slide histology images to improve classification performance. Our results show that HistoPerm consistently improves patch- and slide-level classification performance in terms of accuracy, F1-score, and AUC.
  arXiv  Detail & Related papers  (2022-09-13T17:35:08Z)
• Cross-Site Severity Assessment of COVID-19 from CT Images via Domain Adaptation [64.59521853145368]
  Early and accurate severity assessment of Coronavirus disease 2019 (COVID-19) based on computed tomography (CT) images offers a great help to the estimation of intensive care unit event. To augment the labeled data and improve the generalization ability of the classification model, it is necessary to aggregate data from multiple sites. This task faces several challenges including class imbalance between mild and severe infections, domain distribution discrepancy between sites, and presence of heterogeneous features.
  arXiv  Detail & Related papers  (2021-09-08T07:56:51Z)
• An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization [45.00998416720726]
  We propose a framework to address the unique properties of medical images. This model first uses a low-capacity, yet memory-efficient, network on the whole image to identify the most informative regions. It then applies another higher-capacity network to collect details from chosen regions. Finally, it employs a fusion module that aggregates global and local information to make a final prediction.
  arXiv  Detail & Related papers  (2020-02-13T15:28:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.