Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?

• URL: http://arxiv.org/abs/2607.04694v1
• Date: Mon, 06 Jul 2026 05:52:33 GMT
• Title: Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?
• Abstract summary: Raw medical data is often raw, heterogeneous, and fragmented across different sources.<n>Models are evaluated on their ability to identify source formats, convert raw medical images into VLM-compatible visual inputs, extract relevant textual information, and organize the results into structured image-text pairs.<n>Our research highlights raw medical data standardization as a critical bottleneck for medical AI diagnosis in real practice.
• Score: 22.530682215493798
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As vision-language models (VLMs) are increasingly applied to medical AI, existing benchmarks mainly focus on evaluating their diagnosis ability over given medical images and texts, implicitly assuming that standardized medical images, texts or question-answer pairs are already prepared. However, this assumption does not hold when we apply VLMs in real clinical practice, where medical data is often raw, heterogeneous, and fragmented across different sources. In this paper, we study this missing step, i.e., raw medical data standardization. Specifically, models are given raw dataset folders and evaluated on their ability to identify source formats, convert raw medical images into VLM-compatible visual inputs, extract relevant textual information, and organize the results into structured image-text pairs. To construct this Medical Data Standardization Benchmark (MDS-Bench), we manually annotate 1,939 raw medical data standardization tasks covering diverse clinical practice, radiology modalities, annotation formats, and directory layouts. Extensive experiments show that even the best performing VLMs, i.e., Gemini 3 Flash, achieve only 48.6% end-to-end success rate. Our research highlights raw medical data standardization as a critical bottleneck for medical AI diagnosis in real practice.

Related papers

• MetaDent: Labeling Clinical Images for Vision-Language Models in Dentistry [32.93319761809706]
  Vision-Language Models (VLMs) have demonstrated significant potential in medical image analysis.<n>Their application in intraoral photography remains largely underexplored due to the lack of fine-grained, annotated datasets and comprehensive benchmarks.<n>We present MetaDent, a comprehensive resource that includes a novel and large-scale dentistry image dataset collected from clinical, public, and web sources.
  arXiv  Detail & Related papers  (2026-04-16T10:56:54Z)
• MedROV: Towards Real-Time Open-Vocabulary Detection Across Diverse Medical Imaging Modalities [89.81463562506637]
  We introduce MedROV, the first Real-time Open Vocabulary detection model for medical imaging.<n>By leveraging contrastive learning and cross-modal representations, MedROV effectively detects both known and novel structures.
  arXiv  Detail & Related papers  (2025-11-25T18:59:53Z)
• TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models [54.48710348910535]
  Existing medical reasoning benchmarks primarily focus on analyzing a patient's condition based on an image from a single visit.<n>We introduce TemMed-Bench, the first benchmark designed for analyzing changes in patients' conditions between different clinical visits.
  arXiv  Detail & Related papers  (2025-09-29T17:51:26Z)
• On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI [4.866086225040713]
  We introduce a perturbation-based approach to quantify a model's reliance on each modality in binary classification tasks.<n>By swapping images or text between samples with opposing labels, we expose modality-specific biases.
  arXiv  Detail & Related papers  (2025-07-31T21:35:52Z)
• Towards Domain Specification of Embedding Models in Medicine [1.0713888959520208]
  We propose a comprehensive benchmark suite of 51 tasks spanning classification, clustering, pair classification, and retrieval modeled on the Massive Text Embedding Benchmark (MTEB)<n>Our results demonstrate that this combined approach establishes a robust evaluation framework and yields embeddings that consistently outperform state of the art alternatives in different tasks.
  arXiv  Detail & Related papers  (2025-07-25T16:15:00Z)
• TCM-Tongue: A Standardized Tongue Image Dataset with Pathological Annotations for AI-Assisted TCM Diagnosis [12.39302160184597]
  Traditional Chinese medicine (TCM) tongue diagnosis faces standardization challenges due to subjective interpretation and inconsistent imaging protocols.<n>To address this gap, we present the first specialized dataset for AI-driven TCM tongue diagnosis.<n>The dataset includes 6,719 high-quality images captured under standardized conditions and annotated with 20 pathological symptom categories.
  arXiv  Detail & Related papers  (2025-07-24T10:49:31Z)
• UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities [68.12889379702824]
  Vision-Language Models (VLMs) trained via contrastive learning have achieved notable success in natural image tasks.<n>UniMed is a large-scale, open-source multi-modal medical dataset comprising over 5.3 million image-text pairs.<n>We trained UniMed-CLIP, a unified VLM for six modalities, achieving notable gains in zero-shot evaluations.
  arXiv  Detail & Related papers  (2024-12-13T18:59:40Z)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI [67.09501109871351]
  Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals. GMAI-MMBench is the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format.
  arXiv  Detail & Related papers  (2024-08-06T17:59:21Z)
• Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
  We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
  arXiv  Detail & Related papers  (2024-04-27T05:03:42Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• MedFMC: A Real-world Dataset and Benchmark For Foundation Model Adaptation in Medical Image Classification [41.16626194300303]
  Foundation models, often pre-trained with large-scale data, have achieved paramount success in jump-starting various vision and language applications. Recent advances further enable adapting foundation models in downstream tasks efficiently using only a few training samples. Yet, the application of such learning paradigms in medical image analysis remains scarce due to the shortage of publicly accessible data and benchmarks.
  arXiv  Detail & Related papers  (2023-06-16T01:46:07Z)
• Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime [70.04389979779195]
  This paper explores training medical vision-language models (VLMs) where the visual and language inputs are embedded into a common space. We explore several candidate methods to improve low-data performance, including adapting generic pre-trained models to novel image and text domains. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports.
  arXiv  Detail & Related papers  (2023-03-30T18:20:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.