Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography
Abstract Overview
This paper presents CT-DiagVLM, a vision-language pretraining framework designed for 3D chest CT, motivated by limitations of prior methods in volumetric visual encoding and coarse image-text alignment. The method combines a CNN-ViT hybrid encoder for efficient multi-scale 3D representation, disease-level contrastive learning that extracts condition-specific semantics from reports via learnable query tokens, and a diagnosis-aware prompting strategy based on real clinical phrases and aggregated disease prototypes. The framework is evaluated on zero-shot diagnosis for standard 16-disease benchmarks, a larger 60-disease setting built with LLM-generated pseudo-labels, and radiology report generation. Across these settings, the evidence indicates improved diagnostic discrimination, external generalization, and transfer to downstream report generation.
Novelty
The main novelty is a disease-centric vision-language pretraining design for 3D CT that moves beyond global or organ-level alignment to explicit disease-level alignment within organs. It also introduces a hybrid 3D CNN-ViT encoder that preserves compatibility with pretrained vision-language priors while capturing local volumetric anatomy, along with a prompt strategy that replaces handcrafted templates with prototypes derived from real clinical report language.
Results
On zero-shot diagnosis, CT-DiagVLM achieves 84.4% AUC on CT-RATE and 75.4% AUC on the external Rad-ChestCT set, reported as improvements over prior methods and over the authors' strengthened base model. In the 60-disease pseudo-labeled setting, it reaches 85.6% AUC versus 75.8% for the base model, though the paper notes a sensitivity decrease alongside higher specificity. The pretrained representation also transfers to report generation, where the model reports F1 scores of 45.5% on CT-RATE and 37.3% on Rad-ChestCT.
Key Points
- The proposed CNN-ViT hybrid encoder replaces standard ViT patch embedding with a 3D CNN and multi-scale fusion, aiming to retain local anatomical detail while remaining compatible with pretrained transformer priors.
- Disease-level contrastive learning uses learnable condition queries to extract disease-specific semantics from full radiology reports and align them with organ-level visual features, addressing conflation of multiple pathologies within the same organ.
- A diagnosis-aware prompt strategy builds disease prototypes from many real report descriptions rather than fixed templates, and the experiments show gains in zero-shot diagnosis, scaling to 60 diseases, and downstream report generation.