MedVTok: A General-Purpose Medical Visual Tokenizer
Abstract Overview
MedVTok is a general-purpose medical visual tokenizer designed to map both 2D images and 3D volumes from diverse imaging modalities into a unified token space. The method addresses two stated challenges in unified medical tokenization: preserving volumetric anatomical continuity and capturing clinical semantics across heterogeneous modalities. To do this, it combines Slice Differential Regularization for inter-slice consistency in 3D data with Multi-Expert Representation Alignment that leverages multiple modality-specific expert encoders. The model is trained on 28,769,502 images and 70,144 volumes spanning 9 modalities, and is evaluated across 24 benchmarks covering classification, retrieval, segmentation, synthesis, visual question answering, and medical report generation.
Novelty
The paper’s main novelty is a single tokenizer that natively supports both 2D and 3D medical imaging while aiming to unify representation across modalities and downstream tasks. Its distinctive technical contributions are Slice Differential Regularization for through-plane anatomical continuity and Multi-Expert Representation Alignment, which integrates multiple expert encoders rather than aligning to a single semantic model.
Results
Across the reported evaluations, MedVTok achieves state-of-the-art or best-reported performance over competing tokenizers on 24 benchmarks. The paper shows best average results in token fidelity, classification, retrieval, segmentation, visual question answering, and medical report generation, and also reports the best average FID for image synthesis among the compared tokenizers.
Key Points
- MedVTok provides a shared token interface for heterogeneous medical imaging, covering both 2D images and 3D volumes across 9 modalities.
- The method combines structural supervision for volumetric continuity with semantic alignment from multiple modality-specific experts.
- Empirical evaluation across 24 benchmarks indicates strong generality across reconstruction and six downstream task families.