BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs

• URL: http://arxiv.org/abs/2604.17629v1
• Date: Sun, 19 Apr 2026 21:54:53 GMT
• Title: BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
• Abstract summary: BioVLM is a prompt-learning framework that improves cross-domain generalization without extensive backbone fine-tuning.<n>On 11 MedMNIST+ 2D datasets, BioVLM achieves new state of the art across three distinct generalization settings.
• Score: 38.20973682908207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained biomedical vision-language models (VLMs) such as BioMedCLIP perform well on average but often degrade on challenging modalities where inter-class margins are small and acquisition-specific variations are pronounced, especially under few-shot supervision and when modality priors differ from pretraining corpora substantially. We propose BioVLM, a prompt-learning framework that improves cross-domain generalization without extensive backbone fine-tuning. BioVLM learns a diverse prompt bank and introduces dynamic prompt selection: for each input, it selects the most discriminative prompts via a low-entropy criterion on the predictive distribution, effectively coupling sparse few-shot evidence with rich LLM semantic priors. To strengthen this coupling, we distill high-confidence LLM-derived attributes and enforce robust knowledge transfer through strong/weak augmentation consistency. At test time, BioVLM adapts by choosing modality-appropriate prompts, enabling transfer to unseen categories and domains, while keeping training lightweight and inference efficient. On 11 MedMNIST+ 2D datasets, BioVLM achieves new state of the art across three distinct generalization settings. Codes are available at https://github.com/mainaksingha01/BioVLM.

Related papers

• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification [59.24009931000134]
  Multi-View Synergistic Learning (MVSL) is a unified framework that addresses adaptation paradigms, representation granularity, and disease semantic relationships.<n>MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics.<n>It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence.<n>MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
  arXiv  Detail & Related papers  (2026-04-27T02:41:27Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs [7.668768558131368]
  We propose vMFCoOp, a framework that inversely estimates von Mises-Fisher (vMF) distributions on a shared Hyperspherical Manifold.<n> vMFCoOp demonstrates consistent improvements across 14 medical datasets, 12 medical imaging modalities, and 13 anatomical regions, outperforming state-of-the-art methods in accuracy, generalization, and clinical applicability.
  arXiv  Detail & Related papers  (2025-11-12T18:38:33Z)
• BioVERSE: Representation Alignment of Biomedical Modalities to LLMs for Multi-Modal Reasoning [0.36855563110245826]
  We present BIOVERSE, a two-stage approach that adapts pretrained BioFMs as modality encoders.<n>The approach first aligns each modality to a shared LLM space.<n>It then applies standard instruction tuning with multi-modal data to bring them together for downstream reasoning.
  arXiv  Detail & Related papers  (2025-10-01T20:07:36Z)
• MuSACo: Multimodal Subject-Specific Selection and Adaptation for Expression Recognition with Co-Training [52.99217736494484]
  We introduce MuSACo, a multi-modal subject-specific selection and adaptation method for personalized expression recognition.<n>This makes MuSACo relevant for affective computing applications in digital health, such as patient-specific assessment for stress or pain.<n>Our experimental results on challenging multimodal ER datasets: BioVid and StressID, show that MuSACo can outperform UDA (blending) and state-of-the-art MSDA methods.
  arXiv  Detail & Related papers  (2025-08-17T23:08:21Z)
• Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models [55.74944165932666]
  We introduce Biology-Instructions, the first large-scale instruction-tuning dataset for multi-omics biological sequences.<n>This dataset bridges large language models (LLMs) and complex biological sequence-related tasks, enhancing their versatility and reasoning.<n>We also highlight significant limitations of current state-of-the-art LLMs on multi-omics tasks without specialized training.
  arXiv  Detail & Related papers  (2024-12-26T12:12:23Z)
• MVKTrans: Multi-View Knowledge Transfer for Robust Multiomics Classification [14.533025681231294]
  We propose the multi-view knowledge transfer learning framework, which transfers intra- and inter-omics knowledge in an adaptive manner. Specifically, we design a graph contrastive module that is trained on unlabeled data to effectively learn and transfer the underlying intra-omics patterns to the supervised task. In light of the varying discriminative capacities of modalities across different diseases and/or samples, we introduce an adaptive and bi-directional cross-omics distillation module.
  arXiv  Detail & Related papers  (2024-11-13T15:45:46Z)
• Improving Biomedical Entity Linking with Retrieval-enhanced Learning [53.24726622142558]
  $k$NN-BioEL provides a BioEL model with the ability to reference similar instances from the entire training corpus as clues for prediction. We show that $k$NN-BioEL outperforms state-of-the-art baselines on several datasets.
  arXiv  Detail & Related papers  (2023-12-15T14:04:23Z)
• Multi-Scale and Multi-Modal Contrastive Learning Network for Biomedical Time Series [3.769171969383224]
  This paper presents a multi-scale and multi-modal biomedical time series representation learning (MBSL) network with contrastive learning to migrate variations. Experiments against four bio-medical applications show that MBSL outperforms state-of-the-art models by 33.9% mean average errors (MAE) in respiration rate, by 13.8% MAE in exercise heart rate, by 1.41% accuracy in human activity recognition, and by 1.14% F1-score in obstructive sleep apnea-hypopnea syndrome.
  arXiv  Detail & Related papers  (2023-12-06T14:33:50Z)
• BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphs [27.32543389443672]
  We present BioBridge, a novel parameter-efficient learning framework to bridge independently trained unimodal FMs to establish multimodal behavior. Our empirical results demonstrate that BioBridge can beat the best baseline KG embedding methods. We also identify BioBridge demonstrates out-of-domain generalization ability by extrapolating to unseen modalities or relations.
  arXiv  Detail & Related papers  (2023-10-05T05:30:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.