MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models

• URL: http://arxiv.org/abs/2607.07673v1
• Date: Wed, 08 Jul 2026 17:26:49 GMT
• Title: MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models
• Abstract summary: We introduce MedPMC, an automated framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models.<n> applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs.<n>Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points.
• Score: 39.13366427393118
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.

Related papers

• A visual large language foundational model for medical image recognition using clinician-oriented social media [21.681667706513423]
  Large language models (LLMs) have demonstrated strong capabilities across diverse domains, showing considerable potential in medicine.<n>Here, we leverage de-identified medical images and expert shared commentaries on clinician-oriented social media.<n>By combining an advanced LLM with clinician-in-the-loop verification, we established a rigorous pipeline to construct ThoughtMed-1M.
  arXiv  Detail & Related papers  (2026-09-07T01:30:34Z)
• ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding [67.57715989349471]
  Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge.<n>We introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding.<n>We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks.
  arXiv  Detail & Related papers  (2026-07-27T17:59:49Z)
• A Vision-language Framework for Comparative Reasoning in Radiology [68.52471827773482]
  We formulate radiological comparison as an entity-aware cross-image reasoning problem.<n>We construct a large-scale comparative imaging resource derived from routine image-report pairs.<n>We develop MedReCo, an entity-aware visual encoder for controllable retrieval of clinically analogous cases.
  arXiv  Detail & Related papers  (2026-06-04T17:12:47Z)
• MedGemma Technical Report [75.88152277443179]
  We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B.<n>MedGemma demonstrates advanced medical understanding and reasoning on images and text.<n>We additionally introduce MedSigLIP, a medically-tuned vision encoder derived from SigLIP.
  arXiv  Detail & Related papers  (2025-07-07T17:01:44Z)
• From Embeddings to Accuracy: Comparing Foundation Models for Radiographic Classification [33.96915720287914]
  We evaluate embeddings from seven foundation models for training lightweight adapters in radiography classification.<n>The combination of MedImageInsight embeddings with an SVM and an adapter achieved the highest mean area under the curve (mAUC) of 93.1%.<n>These lightweight groups are computationally efficient, training in minutes and performing inference in seconds on a CPU, making them practical for clinical use.
  arXiv  Detail & Related papers  (2025-05-16T03:39:46Z)
• Multimodal Medical Disease Classification with LLaMA II [0.14999444543328289]
  We use the text-image pair dataset from OpenI consisting of 2D chest X-rays associated with clinical reports.<n>Our focus is on fusion methods for merging text and vision information extracted from medical datasets.<n>The newly introduced multimodal architecture can be applied to other multimodal datasets with little effort and can be easily adapted for further research.
  arXiv  Detail & Related papers  (2024-12-02T09:18:07Z)
• PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction [4.659998272408215]
  Early detection of a pulmonary embolism (PE) is critical for enhancing patient survival rates. We suggest a multimodal fusion methodology, termed PE-MVCNet, which capitalizes on Computed Tomography Pulmonary Angiography imaging and EMR data. Our proposed model outperforms existing methodologies, corroborating that our multimodal fusion model excels compared to models that use a single data modality.
  arXiv  Detail & Related papers  (2024-02-27T03:53:27Z)
• C^2M-DoT: Cross-modal consistent multi-view medical report generation with domain transfer network [67.97926983664676]
  We propose a cross-modal consistent multi-view medical report generation with a domain transfer network (C2M-DoT) C2M-DoT substantially outperforms state-of-the-art baselines in all metrics.
  arXiv  Detail & Related papers  (2023-10-09T02:31:36Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents [35.64805788623848]
  We build and release PMC-OA, a biomedical dataset with 1.6M image-caption pairs collected from PubMedCentral's OpenAccess subset. PMC-OA covers diverse modalities or diseases, with majority of the image-caption samples aligned at finer-grained level. While pretraining a CLIP-style model on PMC-OA, our model named PMC-CLIP achieves state-of-the-art results on various downstream tasks.
  arXiv  Detail & Related papers  (2023-03-13T16:13:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.