Evaluating Vision Foundation Models for Pixel and Object Classification in Microscopy

• URL: http://arxiv.org/abs/2603.19802v1
• Date: Fri, 20 Mar 2026 09:40:41 GMT
• Title: Evaluating Vision Foundation Models for Pixel and Object Classification in Microscopy
• Abstract summary: We investigate whether vision foundation models (VFMs) can improve pixel and object classification compared to current approaches.<n>Our results demonstrate consistent improvements over hand-crafted features.<n>Our study establishes a benchmark for VFMs in microscopy and informs future developments in this area.
• Score: 1.0848249195108657
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning underlies most modern approaches and tools in computer vision, including biomedical imaging. However, for interactive semantic segmentation (often called pixel classification in this context) and interactive object-level classification (object classification), feature-based shallow learning remains widely used. This is due to the diversity of data in this domain, the lack of large pretraining datasets, and the need for computational and label efficiency. In contrast, state-of-the-art tools for many other vision tasks in microscopy - most notably cellular instance segmentation - already rely on deep learning and have recently benefited substantially from vision foundation models (VFMs), particularly SAM. Here, we investigate whether VFMs can also improve pixel and object classification compared to current approaches. To this end, we evaluate several VFMs, including general-purpose models (SAM, SAM2, DINOv3) and domain-specific ones ($μ$SAM, PathoSAM), in combination with shallow learning and attentive probing on five diverse and challenging datasets. Our results demonstrate consistent improvements over hand-crafted features and provide a clear pathway toward practical improvements. Furthermore, our study establishes a benchmark for VFMs in microscopy and informs future developments in this area.

Related papers

• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• Toward a Multi-View Brain Network Foundation Model: Cross-View Consistency Learning Across Arbitrary Atlases [62.33465338932216]
  MV-BrainFM is a multi-view brain network foundation model designed to learn generalizable and scalable representations from brain networks constructed with arbitrary atlases.<n>Experiments on more than 20K subjects from 17 fMRI datasets show that MV-BrainFM consistently outperforms 14 existing brain network foundation models.
  arXiv  Detail & Related papers  (2026-03-20T11:55:00Z)
• Seeing Further on the Shoulders of Giants: Knowledge Inheritance for Vision Foundation Models [54.517276878748305]
  Vision foundation models (VFMs) are predominantly developed using data-centric methods.<n>Many open-source vision models have been pretrained on domain-specific data.<n>We present a new model-driven approach for training VFMs through joint knowledge transfer and preservation.
  arXiv  Detail & Related papers  (2025-08-20T13:30:23Z)
• Benchmarking histopathology foundation models in a multi-center dataset for skin cancer subtyping [1.927195358774599]
  Pretraining on large-scale, in-domain datasets grants histopathology foundation models (FM) the ability to learn task-agnostic data representations.<n>In computational pathology, automated whole slide image analysis requires multiple instance learning (MIL) frameworks due to the gigapixel scale of the slides.<n>Our work presents a novel benchmark for evaluating histopathology FMs as patch-level feature extractors within a MIL classification framework.
  arXiv  Detail & Related papers  (2025-06-23T14:12:16Z)
• Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology [2.7280901660033643]
  This work explores the scaling properties of weakly supervised classifiers and self-supervised masked autoencoders (MAEs) Our results show that ViT-based MAEs outperform weakly supervised classifiers on a variety of tasks, achieving as much as a 11.5% relative improvement when recalling known biological relationships curated from public databases. We develop a new channel-agnostic MAE architecture (CA-MAE) that allows for inputting images of different numbers and orders of channels at inference time.
  arXiv  Detail & Related papers  (2024-04-16T02:42:06Z)
• A Novel Benchmark for Few-Shot Semantic Segmentation in the Era of Foundation Models [7.428199805959228]
  Few-shot semantic segmentation (FSS) is a crucial challenge in computer vision.<n>With the emergence of vision foundation models (VFM) as generalist feature extractors, we seek to explore the adaptation of these models for FSS.<n>We propose a novel realistic benchmark with a simple and straightforward adaptation process tailored for this task.
  arXiv  Detail & Related papers  (2024-01-20T19:50:51Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)
• Domain Generalization for Mammographic Image Analysis with Contrastive Learning [62.25104935889111]
  The training of an efficacious deep learning model requires large data with diverse styles and qualities. A novel contrastive learning is developed to equip the deep learning models with better style generalization capability. The proposed method has been evaluated extensively and rigorously with mammograms from various vendor style domains and several public datasets.
  arXiv  Detail & Related papers  (2023-04-20T11:40:21Z)
• Understanding the Tricks of Deep Learning in Medical Image Segmentation: Challenges and Future Directions [66.40971096248946]
  In this paper, we collect a series of MedISeg tricks for different model implementation phases. We experimentally explore the effectiveness of these tricks on consistent baselines. We also open-sourced a strong MedISeg repository, where each component has the advantage of plug-and-play.
  arXiv  Detail & Related papers  (2022-09-21T12:30:05Z)
• Segmentation in large-scale cellular electron microscopy with deep learning: A literature survey [6.144134660210243]
  deep learning algorithms achieved impressive results in both pixel-level labeling and the labeling of separate instances of the same class. In this review, we examine how these algorithms were adapted to the task of segmenting cellular and sub-cellular structures in EM images.
  arXiv  Detail & Related papers  (2022-06-14T20:57:49Z)
• Digital Fingerprinting of Microstructures [44.139970905896504]
  Finding efficient means of fingerprinting microstructural information is a critical step towards harnessing data-centric machine learning approaches. Here, we consider microstructure classification and utilise the resulting features over a range of related machine learning tasks. In particular, methods that leverage transfer learning with convolutional neural networks (CNNs), pretrained on the ImageNet dataset, are generally shown to outperform other methods.
  arXiv  Detail & Related papers  (2022-03-25T15:40:44Z)
• Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types [50.1843146606122]
  A simple form of transfer learning is common in current state-of-the-art computer vision models. Previous systematic studies of transfer learning have been limited and the circumstances in which it is expected to work are not fully understood. In this paper we carry out an extensive experimental exploration of transfer learning across vastly different image domains.
  arXiv  Detail & Related papers  (2021-03-24T16:24:20Z)
• Multimodal Transfer Learning-based Approaches for Retinal Vascular Segmentation [2.672151045393935]
  The study of the retinal microcirculation is a key issue in the analysis of many ocular and systemic diseases, like hypertension or diabetes. FCNs usually represent the most successful approach to image segmentation. In this work, we present multimodal transfer learning-based approaches for retinal vascular segmentation.
  arXiv  Detail & Related papers  (2020-12-18T10:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.