Are General-Purpose Vision Models All We Need for 2D Medical Image Segmentation? A Cross-Dataset Empirical Study

• URL: http://arxiv.org/abs/2603.13044v1
• Date: Fri, 13 Mar 2026 14:52:51 GMT
• Title: Are General-Purpose Vision Models All We Need for 2D Medical Image Segmentation? A Cross-Dataset Empirical Study
• Abstract summary: We compare specialized medical segmentation architectures (SMAs) and general-purpose vision models (GP-VMs) for 2D medical images.<n>For the analyzed datasets, GP-VMs out-perform the majority of specialized MIS models.<n>XAI analyses indicate that GP-VMs can capture clinically relevant structures without explicit domain-specific architectural design.
• Score: 1.0898188221663585
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Medical image segmentation (MIS) is a fundamental component of computer-assisted diagnosis and clinical decision support systems. Over the past decade, numerous architectures specifically tailored to medical imaging have emerged to address domain-specific challenges such as low contrast, small anatomical structures, and limited annotated data. In parallel, rapid progress in computer vision has produced highly capable general-purpose vision models (GP-VMs) originally designed for natural images. Despite their strong performance on standard vision benchmarks, their effectiveness for MIS remains insufficiently understood. In this work, we conduct a controlled empirical study to examine whether specialized medical segmentation architectures (SMAs) provide systematic advantages over modern GP-VMs for 2D MIS. We compare eleven SMAs and GP-VMs using a unified training and evaluation protocol. Experiments are performed across three heterogeneous datasets covering different imaging modalities, class structures, and data characteristics. Beyond segmentation accuracy, we analyze qualitative Grad-CAM visualizations to investigate explainability (XAI) behavior. Our results demonstrate that, for the analyzed datasets, GP-VMs out-perform the majority of specialized MIS models. Moreover, XAI analyses indicate that GP-VMs can capture clinically relevant structures without explicit domain-specific architectural design. These findings suggest that GP-VMs can represent a viable alternative to domain-specific methods, highlighting the importance of informed model selection for end-to-end MIS systems. All code and resources are available at GitHub.

Related papers

• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• Vision Foundry: A System for Training Foundational Vision AI Models [0.0]
  Vision Foundry is a code-free, HIPAA-compliant platform that democratizes pre-training, adaptation, and deployment of vision models.<n>By bridging the gap between advanced representation learning and practical application, Vision Foundry enables domain experts to develop state-of-the-art clinical AI tools.
  arXiv  Detail & Related papers  (2025-12-03T14:02:22Z)
• Foundation Models in Medical Image Analysis: A Systematic Review and Meta-Analysis [7.905460364844281]
  Foundations models (FMs) have revolutionized medical image analysis, demonstrating strong zero- and few-shot performance across diverse medical imaging tasks.<n>FMs leverage large corpora of labeled and unlabeled multimodal datasets to learn generalized representations.<n>Despite the rapid proliferation of FM research in medical imaging, the field remains fragmented.<n>This review article provides a comprehensive and structured analysis of FMs in medical image analysis.
  arXiv  Detail & Related papers  (2025-10-19T19:19:23Z)
• Does DINOv3 Set a New Medical Vision Standard? [67.33543059306938]
  This report investigates whether DINOv3 can serve as a powerful unified encoder for medical vision tasks without domain-specific pre-training.<n>We benchmark DINOv3 across common medical vision tasks, including 2D/3D classification and segmentation.<n>Remarkably, it can even outperform medical-specific foundation models like BiomedCLIP and CT-Net on several tasks.
  arXiv  Detail & Related papers  (2025-09-08T09:28:57Z)
• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation [56.52520416420957]
  We propose Multimodal Causal-Driven Representation Learning (MCDRL) to tackle domain generalization in medical image segmentation.<n>MCDRL consistently outperforms competing methods, yielding superior segmentation accuracy and exhibiting robust generalizability.
  arXiv  Detail & Related papers  (2025-08-07T03:41:41Z)
• MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis [4.361234801854702]
  MIRAGE is a novel FM for the analysis of OCT and scanning laser ophthalmoscopy (SLO) images.<n>We propose a new evaluation benchmark with OCT/SLO classification and segmentation tasks.<n>The comparison with general and specialized FMs and segmentation methods shows the superiority of MIRAGE in both types of tasks.
  arXiv  Detail & Related papers  (2025-06-10T15:25:55Z)
• U-Net in Medical Image Segmentation: A Review of Its Applications Across Modalities [0.0]
  Recent advancements in Artificial Intelligence (AI) and Deep Learning (DL) have transformed medical image segmentation (MIS)<n>These models enable efficient, precise pixel-wise classification across various imaging modalities.<n>This review explores various medical imaging techniques, examines the U-Net architectures and their adaptations, and discusses their application across different modalities.
  arXiv  Detail & Related papers  (2024-12-03T08:11:06Z)
• A Survey of Medical Vision-and-Language Applications and Their Techniques [48.268198631277315]
  Medical vision-and-language models (MVLMs) have attracted substantial interest due to their capability to offer a natural language interface for interpreting complex medical data. Here, we provide a comprehensive overview of MVLMs and the various medical tasks to which they have been applied. We also examine the datasets used for these tasks and compare the performance of different models based on standardized evaluation metrics.
  arXiv  Detail & Related papers  (2024-11-19T03:27:05Z)
• Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation? [10.20366295974822]
  We introduce a novel decode head architecture, HQHSAM, which simply integrates elements from two state-of-the-art decoder heads, HSAM and HQSAM, to enhance segmentation performance. Our experiments on multiple datasets, encompassing various anatomies and modalities, reveal that FMs, particularly with the HQHSAM decode head, improve domain generalization for medical image segmentation.
  arXiv  Detail & Related papers  (2024-09-12T11:41:35Z)
• VIS-MAE: An Efficient Self-supervised Learning Approach on Medical Image Segmentation and Classification [33.699424327366856]
  We present VIsualization and Masked AutoEncoder (VIS-MAE), novel model weights specifically designed for medical imaging.<n>VIS-MAE is trained on a dataset of 2.5 million unlabeled images from various modalities.<n>It is then adapted to classification and segmentation tasks using explicit labels.
  arXiv  Detail & Related papers  (2024-02-01T21:45:12Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• Artificial General Intelligence for Medical Imaging Analysis [92.3940918983821]
  Large-scale Artificial General Intelligence (AGI) models have achieved unprecedented success in a variety of general domain tasks. These models face notable challenges arising from the medical field's inherent complexities and unique characteristics. This review aims to offer insights into the future implications of AGI in medical imaging, healthcare, and beyond.
  arXiv  Detail & Related papers  (2023-06-08T18:04:13Z)
• Domain Shift in Computer Vision models for MRI data analysis: An Overview [64.69150970967524]
  Machine learning and computer vision methods are showing good performance in medical imagery analysis. Yet only a few applications are now in clinical use. Poor transferability of themodels to data from different sources or acquisition domains is one of the reasons for that.
  arXiv  Detail & Related papers  (2020-10-14T16:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.