Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement

• URL: http://arxiv.org/abs/2603.28027v1
• Date: Mon, 30 Mar 2026 04:39:07 GMT
• Title: Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement
• Abstract summary: We propose a parameter-efficient fine-tuning framework, named Cooperative Fine-Grained Refinement of SAM.<n>These three components work cooperatively to enhance local perception, preserve spatial details, and refine boundaries, enabling SAM to perform accurate nuclei instance segmentation directly.
• Score: 24.433000105666114
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Nuclei instance segmentation is critical in computational pathology for cancer diagnosis and prognosis. Recently, the Segment Anything Model has demonstrated exceptional performance in various segmentation tasks, leveraging its rich priors and powerful global context modeling capabilities derived from large-scale pre-training on natural images. However, directly applying SAM to the medical imaging domain faces significant limitations: it lacks sufficient perception of the local structural features that are crucial for nuclei segmentation, and full fine-tuning for downstream tasks requires substantial computational costs. To efficiently transfer SAM's robust prior knowledge to nuclei instance segmentation while supplementing its task-aware local perception, we propose a parameter-efficient fine-tuning framework, named Cooperative Fine-Grained Refinement of SAM, consisting of three core components: 1) a Multi-scale Adaptive Local-aware Adapter, which enables effective capability transfer by augmenting the frozen SAM backbone with minimal parameters and instilling a powerful perception of local structures through dynamically generated, multi-scale convolutional kernels; 2) a Hierarchical Modulated Fusion Module, which dynamically aggregates multi-level encoder features to preserve fine-grained spatial details; and 3) a Boundary-Guided Mask Refinement, which integrates multi-context boundary cues with semantic features through explicit supervision, producing a boundary-focused signal to refine initial mask predictions for sharper delineation. These three components work cooperatively to enhance local perception, preserve spatial details, and refine boundaries, enabling SAM to perform accurate nuclei instance segmentation directly.

Related papers

• Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation [10.131681447132644]
  We propose HPMA, a Hierarchical Prototype-Memory Adaptation framework for surgical instrument segmentation.<n>Specifically, HPMA constructs a frozen, multi-scale visual prototype memory bank from annotated surgical scenes.<n>Our approach achieves state-of-the-art performance, outperforming existing foundation model adaptation methods.
  arXiv  Detail & Related papers  (2026-08-25T13:27:57Z)
• When Simplicity Wins: Bottleneck-Aware Context Modeling for Lightweight Semantic Segmentation [58.325533783810016]
  We propose SiConMo, a lightweight yet effective framework for semantic segmentation.<n>We show that simplicity arises from a key design principle: at very low computational budgets, the bottleneck is the most efficient stage to integrate local and global context.<n> experiments on ADE20K, PASCAL Context, Cityscapes, and COCO-Stuff demonstrate that SiConMo achieves a state-of-the-art accuracy-efficiency trade-off.
  arXiv  Detail & Related papers  (2026-08-19T14:47:07Z)
• Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing [0.42970700836450487]
  We present a comprehensive, multi-task empirical evaluation across remote sensing scene classification, object detection, and instance segmentation.<n>Our results demonstrate that SAM 3 avoids the overfitting commonly seen in legacy domain-adapted models.<n>It remains fundamentally constrained by sub-pixel resolution limits and overhead semantic blind spots.
  arXiv  Detail & Related papers  (2026-07-10T16:33:05Z)
• Exploring SAM Supervision for Fine-Grained UAV Target Segmentation under Data Scarcity [0.0]
  Unmanned aerial vehicle (UAV) target segmentation is challenging due to the small size of objects, appearance variations, cluttered backgrounds, and the scarcity of densely annotated data.<n>This paper investigates the use of SAM3 as a pseudo-label generator for training compact segmentation networks.
  arXiv  Detail & Related papers  (2026-07-04T07:59:10Z)
• TGC-Net: A Structure-Aware and Semantically-Aligned Framework for Text-Guided Medical Image Segmentation [56.09179939570486]
  We propose TGC-Net, a CLIP-based framework focusing on parameter-efficient, task-specific adaptations.<n>TGC-Net achieves state-of-the-art performance with substantially fewer trainable parameters, including notable Dice gains on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-12-24T12:06:26Z)
• Decoding with Structured Awareness: Integrating Directional, Frequency-Spatial, and Structural Attention for Medical Image Segmentation [6.82200201381917]
  This paper proposes a novel decoder framework specifically designed for medical image segmentation, comprising three core modules.<n>First, the Adaptive Cross-Fusion Attention (ACFA) module integrates channel feature enhancement with spatial attention mechanisms to enhance responsiveness to key regions and structural orientations.<n>Second, the Triple Feature Fusion Attention (TFFA) module fuses features from Spatial, Fourier and Wavelet domains, achieving joint frequency-spatial representation while preserving local information such as edges and textures.<n>Third, the Structural-aware Multi-scale Masking Module (SMMM) optimize the skip connections between encoder and decoder by leveraging multi-scale context and structural s
  arXiv  Detail & Related papers  (2025-12-05T07:39:14Z)
• VesSAM: Efficient Multi-Prompting for Segmenting Complex Vessel [68.24765319399286]
  We present VesSAM, a powerful and efficient framework tailored for 2D vessel segmentation.<n>VesSAM integrates (1) a convolutional adapter to enhance local texture features, (2) a multi-prompt encoder that fuses anatomical prompts, and (3) a lightweight mask decoder to reduce jagged artifacts.<n>VesSAM consistently outperforms state-of-the-art PEFT-based SAM variants by over 10% Dice and 13% IoU.
  arXiv  Detail & Related papers  (2025-11-02T15:47:05Z)
• SAM4EM: Efficient memory-based two stage prompt-free segment anything model adapter for complex 3D neuroscience electron microscopy stacks [6.277236040603983]
  We present SAM4EM, a novel approach for 3D segmentation of complex neural structures in electron microscopy (EM) data.<n>Our contributions include the development of a prompt-free adapter for SAM using two stage mask decoding.<n>We release a unique benchmark dataset for the segmentation of astrocytic processes and synapses.
  arXiv  Detail & Related papers  (2025-04-30T11:38:02Z)
• CFMD: Dynamic Cross-layer Feature Fusion for Salient Object Detection [7.262250906929891]
  Cross-layer feature pyramid networks (CFPNs) have achieved notable progress in multi-scale feature fusion and boundary detail preservation for salient object detection.<n>To address these challenges, we propose CFMD, a novel cross-layer feature pyramid network that introduces two key innovations.<n>First, we design a context-aware feature aggregation module (CFLMA), which incorporates the state-of-the-art Mamba architecture to construct a dynamic weight distribution mechanism.<n>Second, we introduce an adaptive dynamic upsampling unit (CFLMD) that preserves spatial details during resolution recovery.
  arXiv  Detail & Related papers  (2025-04-02T03:22:36Z)
• DSU-Net:An Improved U-Net Model Based on DINOv2 and SAM2 with Multi-scale Cross-model Feature Enhancement [7.9006143460465355]
  This paper proposes a multi-scale feature collabora-tion framework guided by DINOv2 for SAM2, with core innovations in three aspects.<n>It surpasses existing state-of-the-art meth-ods in downstream tasks such as camouflage target detection and salient ob-ject detection, without requiring costly training processes.
  arXiv  Detail & Related papers  (2025-03-27T06:08:24Z)
• Adapting Segment Anything Model for Unseen Object Instance Segmentation [70.60171342436092]
  Unseen Object Instance (UOIS) is crucial for autonomous robots operating in unstructured environments. We propose UOIS-SAM, a data-efficient solution for the UOIS task. UOIS-SAM integrates two key components: (i) a Heatmap-based Prompt Generator (HPG) to generate class-agnostic point prompts with precise foreground prediction, and (ii) a Hierarchical Discrimination Network (HDNet) that adapts SAM's mask decoder.
  arXiv  Detail & Related papers  (2024-09-23T19:05:50Z)
• ASPS: Augmented Segment Anything Model for Polyp Segmentation [77.25557224490075]
  The Segment Anything Model (SAM) has introduced unprecedented potential for polyp segmentation. SAM's Transformer-based structure prioritizes global and low-frequency information. CFA integrates a trainable CNN encoder branch with a frozen ViT encoder, enabling the integration of domain-specific knowledge.
  arXiv  Detail & Related papers  (2024-06-30T14:55:32Z)
• Perspective+ Unet: Enhancing Segmentation with Bi-Path Fusion and Efficient Non-Local Attention for Superior Receptive Fields [19.71033340093199]
  We propose a novel architecture, Perspective+ Unet, to overcome limitations in medical image segmentation. The framework incorporates an efficient non-local transformer block, named ENLTB, which utilizes kernel function approximation for effective long-range dependency capture. Experimental results on the ACDC and datasets demonstrate the effectiveness of our proposed Perspective+ Unet.
  arXiv  Detail & Related papers  (2024-06-20T07:17:39Z)
• Learning Multiscale Consistency for Self-supervised Electron Microscopy Instance Segmentation [48.267001230607306]
  We propose a pretraining framework that enhances multiscale consistency in EM volumes. Our approach leverages a Siamese network architecture, integrating strong and weak data augmentations. It effectively captures voxel and feature consistency, showing promise for learning transferable representations for EM analysis.
  arXiv  Detail & Related papers  (2023-08-19T05:49:13Z)
• Unsupervised Instance Segmentation in Microscopy Images via Panoptic Domain Adaptation and Task Re-weighting [86.33696045574692]
  We propose a Cycle Consistency Panoptic Domain Adaptive Mask R-CNN (CyC-PDAM) architecture for unsupervised nuclei segmentation in histopathology images. We first propose a nuclei inpainting mechanism to remove the auxiliary generated objects in the synthesized images. Secondly, a semantic branch with a domain discriminator is designed to achieve panoptic-level domain adaptation.
  arXiv  Detail & Related papers  (2020-05-05T11:08:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.