MedFoundationHub: A Lightweight and Secure Toolkit for Deploying Medical Vision Language Foundation Models

• URL: http://arxiv.org/abs/2508.20345v1
• Date: Thu, 28 Aug 2025 01:39:16 GMT
• Title: MedFoundationHub: A Lightweight and Secure Toolkit for Deploying Medical Vision Language Foundation Models
• Abstract summary: We present MedFoundationHub, a graphical user interface (GUI) toolkit for deploying medical vision-language models.<n>It enables physicians to manually select and use different models without programming expertise.<n>It also ensures privacy-preserving inference through Docker-orchestrated, operating system deployment.
• Score: 9.098674571831088
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in medical vision-language models (VLMs) open up remarkable opportunities for clinical applications such as automated report generation, copilots for physicians, and uncertainty quantification. However, despite their promise, medical VLMs introduce serious security concerns, most notably risks of Protected Health Information (PHI) exposure, data leakage, and vulnerability to cyberthreats - which are especially critical in hospital environments. Even when adopted for research or non-clinical purposes, healthcare organizations must exercise caution and implement safeguards. To address these challenges, we present MedFoundationHub, a graphical user interface (GUI) toolkit that: (1) enables physicians to manually select and use different models without programming expertise, (2) supports engineers in efficiently deploying medical VLMs in a plug-and-play fashion, with seamless integration of Hugging Face open-source models, and (3) ensures privacy-preserving inference through Docker-orchestrated, operating system agnostic deployment. MedFoundationHub requires only an offline local workstation equipped with a single NVIDIA A6000 GPU, making it both secure and accessible within the typical resources of academic research labs. To evaluate current capabilities, we engaged board-certified pathologists to deploy and assess five state-of-the-art VLMs (Google-MedGemma3-4B, Qwen2-VL-7B-Instruct, Qwen2.5-VL-7B-Instruct, and LLaVA-1.5-7B/13B). Expert evaluation covered colon cases and renal cases, yielding 1015 clinician-model scoring events. These assessments revealed recurring limitations, including off-target answers, vague reasoning, and inconsistent pathology terminology.

Related papers

• LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation [38.02853540388593]
  evaluating large language models (LLMs) in medicine is crucial because medical applications require high accuracy with little room for error.<n>We present LLMEval-Med, a new benchmark covering five core medical areas, including 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios.
  arXiv  Detail & Related papers  (2025-06-04T15:43:14Z)
• VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge [36.67150853785481]
  Generalist vision language models (VLMs) have made significant strides in computer vision, but they fall short in specialized fields like healthcare.<n>In traditional computer vision tasks, creative or approximate answers may be acceptable, but in healthcare, precision is paramount.<n>This paper introduces a new framework, VILA-M3, for medical VLMs that utilizes domain knowledge via expert models.
  arXiv  Detail & Related papers  (2024-11-19T22:59:14Z)
• FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injection [83.54960238236548]
  FEDMEKI not only preserves data privacy but also enhances the capability of medical foundation models. FEDMEKI allows medical foundation models to learn from a broader spectrum of medical knowledge without direct data exposure.
  arXiv  Detail & Related papers  (2024-08-17T15:18:56Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• OpenMEDLab: An Open-source Platform for Multi-modality Foundation Models in Medicine [55.29668193415034]
  We present OpenMEDLab, an open-source platform for multi-modality foundation models. It encapsulates solutions of pioneering attempts in prompting and fine-tuning large language and vision models for frontline clinical and bioinformatic applications. It opens access to a group of pre-trained foundation models for various medical image modalities, clinical text, protein engineering, etc.
  arXiv  Detail & Related papers  (2024-02-28T03:51:02Z)
• ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs [48.11532667875847]
  ChatCAD+ is a tool to generate high-quality medical reports and provide reliable medical advice. The Reliable Report Generation module is capable of interpreting medical images and generate high-quality medical reports. The Reliable Interaction module leverages up-to-date information from reputable medical websites to provide reliable medical advice.
  arXiv  Detail & Related papers  (2023-05-25T12:03:31Z)
• MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data [37.60056509129154]
  Large language models (LLMs) hold considerable promise for improving medical, diagnostics, patient care, and education.<n>Yet, there is an urgent need for open-source models that can be deployed on-premises to safeguard patient privacy.<n>We present an innovative dataset consisting of over 160,000 entries, specifically crafted to fine-tune LLMs for effective medical applications.
  arXiv  Detail & Related papers  (2023-04-14T11:28:08Z)
• MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation [110.31526448744096]
  We argue that unlocking this potential requires a systematic way to measure the performance of medical AI models on large-scale heterogeneous data. We are building MedPerf, an open framework for benchmarking machine learning in the medical domain.
  arXiv  Detail & Related papers  (2021-09-29T18:09:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.