UMind-VL: A Generalist Ultrasound Vision-Language Model for Unified Grounded Perception and Comprehensive Interpretation

• URL: http://arxiv.org/abs/2511.22256v1
• Date: Thu, 27 Nov 2025 09:33:00 GMT
• Title: UMind-VL: A Generalist Ultrasound Vision-Language Model for Unified Grounded Perception and Comprehensive Interpretation
• Abstract summary: UMind-VL is a unified foundation model designed to synergize pixel-level structural understanding with complex clinical reasoning.<n>We first introduce UMind-DS, a large-scale multimodal dataset comprising 1.2 million ultrasound image-text pairs across 16 anatomical regions.
• Score: 11.608721320550108
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Despite significant strides in medical foundation models, the ultrasound domain lacks a comprehensive solution capable of bridging low-level Ultrasound Grounded Perception (e.g., segmentation, localization) and high-level Ultrasound Comprehensive Interpretation (e.g., diagnosis, reasoning). To bridge this gap, we propose UMind-VL, a unified foundation model designed to synergize pixel-level structural understanding with complex clinical reasoning. We first introduce UMind-DS, a large-scale multimodal dataset comprising 1.2 million ultrasound image-text pairs across 16 anatomical regions, enriching standard data with pixel-level annotations and clinician-validated rationales. Architecturally, UMind-VL incorporates a lightweight Dynamic Convolutional Mask Decoder that generates masks via dynamic kernels conditioned on LLM outputs. This design, combined with task-specific tokens, unifies segmentation, detection, geometric measurement, and diagnosis tasks within a single framework. Extensive evaluations demonstrate that UMind-VL significantly outperforms existing generalist multimodal models and achieves performance on par with, or superior to, state-of-the-art specialist models across segmentation, detection, keypoint localization, and diagnostic reasoning benchmarks, while maintaining strong generalization ability. We demonstrate the capability of UMind-VL in Figure 1.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment [15.371433625713232]
  EndoVLM is a novel vision-language FM pre-trained on over 348K endoscopic examinations.<n>An Anatomy-Guided Sparse Pooling mechanism utilizes textual descriptions as queries to drive sparse attention.<n>A Progressive Semantic-Aware Alignment strategy models clinical taxonomy (anatomy and pathological status) via structured soft targets.<n>A Semantic-Concentrated Masked Autoencoder is applied exclusively to these semantic-rich frames, integrating low-level visual precision with robust high-level semantic representation.
  arXiv  Detail & Related papers  (2026-08-05T05:56:41Z)
• SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysis [51.81845420301345]
  SonoCLIP is a vision-language foundation model for fetal ultrasound analysis.<n>It integrates segmentation masks as mask-channel visual prompts within the vision encoder.<n>It achieves superior zero-shot transfer performance under both global and mask-guided inference.
  arXiv  Detail & Related papers  (2026-06-28T20:04:49Z)
• Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis [4.767529995968428]
  Digital pathology has fundamentally altered diagnostic by enabling the computational analysis of gigapixel Whole Slide Images (WSIs)<n>We propose the Hierarchical Prototype-based Domain Priors framework, a unified multimodal approach for joint histopathology diagnosis and prognosis.
  arXiv  Detail & Related papers  (2026-04-27T02:51:57Z)
• Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation [59.87131391492088]
  We present Omni-I2C, a benchmark designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executable code.<n>By incorporating authentic user-sourced cases, the benchmark spans a vast spectrum of digital content.<n>Our evaluation reveals a substantial performance gap among leading LMMs.
  arXiv  Detail & Related papers  (2026-03-18T09:10:04Z)
• OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis [53.01523944168442]
  Clinical interpretation relies on both slice-driven local features and volume-driven spatial representations.<n>Existing Large Vision-Language Models (LVLMs) remain fragmented in CT slice versus volumetric understanding.<n>We present OmniCT, a powerful unified slice-volume LVLM for CT scenarios.
  arXiv  Detail & Related papers  (2026-02-18T00:42:41Z)
• HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models [4.099133096025821]
  Vision-language models (VLMs) enable open-ended visual question answering but remain prone to hallucinations.<n>We present HEDGE, a unified framework for hallucination detection that combines controlled visual perturbations, semantic clustering, and robust uncertainty metrics.
  arXiv  Detail & Related papers  (2025-11-16T17:16:31Z)
• RAU: Reference-based Anatomical Understanding with Vision Language Models [26.06602931463068]
  We introduce RAU, a framework for reference-based anatomical understanding with vision-language models (VLMs)<n>We first show that a VLM learns to identify anatomical regions through relative spatial reasoning between reference and target images.<n>Next, we demonstrate that the VLM-derived spatial cues can be seamlessly integrated with the fine-grained segmentation capability of SAM2.
  arXiv  Detail & Related papers  (2025-09-26T14:32:03Z)
• EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound Intelligence [9.731550105507457]
  We propose EchoVLM, a vision-language model specifically designed for ultrasound medical imaging.<n>The model employs a Mixture of Experts (MoE) architecture trained on data spanning seven anatomical regions.<n>EchoVLM achieved significant improvements of 10.15 and 4.77 points in BLEU-1 scores and ROUGE-1 scores respectively.
  arXiv  Detail & Related papers  (2025-09-18T14:07:53Z)
• A Fully Open and Generalizable Foundation Model for Ultrasound Clinical Applications [77.3888788549565]
  We present EchoCare, a novel ultrasound foundation model for generalist clinical use.<n>We developed EchoCare via self-supervised learning on our curated, publicly available, large-scale dataset EchoCareData.<n>With minimal training, EchoCare outperforms state-of-the-art comparison models across 10 representative ultrasound benchmarks.
  arXiv  Detail & Related papers  (2025-09-15T10:05:31Z)
• Large Language Model Evaluated Stand-alone Attention-Assisted Graph Neural Network with Spatial and Structural Information Interaction for Precise Endoscopic Image Segmentation [16.773882069530426]
  We propose FOCUS-Med, which stands for Fusion of spatial and structural graph with attentional context-aware polyp segmentation.<n> FOCUS-Med integrates a Dual Graph Convolutional Network (Dual-GCN) module to capture contextual spatial and topological structural dependencies.<n>Experiments on public benchmarks demonstrate that FOCUS-Med achieves state-of-the-art performance across five key metrics.
  arXiv  Detail & Related papers  (2025-08-09T15:53:19Z)
• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation [56.52520416420957]
  We propose Multimodal Causal-Driven Representation Learning (MCDRL) to tackle domain generalization in medical image segmentation.<n>MCDRL consistently outperforms competing methods, yielding superior segmentation accuracy and exhibiting robust generalizability.
  arXiv  Detail & Related papers  (2025-08-07T03:41:41Z)
• ViKL: A Mammography Interpretation Framework via Multimodal Aggregation of Visual-knowledge-linguistic Features [54.37042005469384]
  We announce MVKL, the first multimodal mammography dataset encompassing multi-view images, detailed manifestations and reports. Based on this dataset, we focus on the challanging task of unsupervised pretraining. We propose ViKL, a framework that synergizes Visual, Knowledge, and Linguistic features.
  arXiv  Detail & Related papers  (2024-09-24T05:01:23Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)
• Towards Robust Partially Supervised Multi-Structure Medical Image Segmentation on Small-Scale Data [123.03252888189546]
  We propose Vicinal Labels Under Uncertainty (VLUU) to bridge the methodological gaps in partially supervised learning (PSL) under data scarcity. Motivated by multi-task learning and vicinal risk minimization, VLUU transforms the partially supervised problem into a fully supervised problem by generating vicinal labels. Our research suggests a new research direction in label-efficient deep learning with partial supervision.
  arXiv  Detail & Related papers  (2020-11-28T16:31:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.