SCISSR: Scribble-Conditioned Interactive Surgical Segmentation and Refinement

• URL: http://arxiv.org/abs/2603.18544v1
• Date: Thu, 19 Mar 2026 07:00:18 GMT
• Title: SCISSR: Scribble-Conditioned Interactive Surgical Segmentation and Refinement
• Abstract summary: We present SCISSR, a scribble-promptable framework for interactive surgical scene segmentation.<n>It introduces a lightweight Scribble that converts freehand scribbles into dense prompt embeddings compatible with the mask decoder.<n> SCISSR achieves 95.41% Dice on EndoVis 2018 with five interaction rounds and 96.30% Dice on CholecSeg8k with three interaction rounds, outperforming iterative point prompting on both benchmarks.
• Score: 6.1665244904503895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate segmentation of tissues and instruments in surgical scenes is annotation-intensive due to irregular shapes, thin structures, specularities, and frequent occlusions. While SAM models support point, box, and mask prompts, points are often too sparse and boxes too coarse to localize such challenging targets. We present SCISSR, a scribble-promptable framework for interactive surgical scene segmentation. It introduces a lightweight Scribble Encoder that converts freehand scribbles into dense prompt embeddings compatible with the mask decoder, enabling iterative refinement for a target object by drawing corrective strokes on error regions. Because all added modules (the Scribble Encoder, Spatial Gated Fusion, and LoRA adapters) interact with the backbone only through its standard embedding interfaces, the framework is not tied to a single model: we build on SAM 2 in this work, yet the same components transfer to other prompt-driven segmentation architectures such as SAM 3 without structural modification. To preserve pre-trained capabilities, we train only these lightweight additions while keeping the remaining backbone frozen. Experiments on EndoVis 2018 demonstrate strong in-domain performance, while evaluation on the out-of-distribution CholecSeg8k further confirms robustness across surgical domains. SCISSR achieves 95.41% Dice on EndoVis 2018 with five interaction rounds and 96.30% Dice on CholecSeg8k with three interaction rounds, outperforming iterative point prompting on both benchmarks.

Related papers

• ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation [19.691062682670378]
  Referring surgical video segmentation requires segmenting a target instrument or tissue region across video frames according to a natural language expression.<n>Recent Segment Anything Model 2 (SAM2) based two-stage methods (e.g., ReSurgSAM2) first ground the referred target in an initial or selected frame, then propagate the selected mask via tracking.<n>We propose Re-Ground-Surg, a lightweight reliability-guided anchor grounding framework to improve SAM2-based referring surgical video segmentation.
  arXiv  Detail & Related papers  (2026-08-25T15:08:24Z)
• Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3) [0.866627581195388]
  We propose a two-stage framework that achieves instance-level segmentation without requiring ground truth masks or manual interaction.<n>In the first stage, we leverage a natural-language-aligned generic prompt - "tool" - to produce binary masks using SAM3's zero-shot capability.<n>In the second stage, these masks are extended to instance-level by integrating a vision-language model (Qwen) that is fine-tuned on SAM3-generated masked regions for instrument classification.
  arXiv  Detail & Related papers  (2026-08-09T18:11:09Z)
• Prompting Diffusion Models for Zero-Shot Instance Segmentation [74.37083707535778]
  We present Prompt2Seg, a spatial conditioning framework for diffusion-based segmentation.<n>Our approach takes spatial prompts, represented as 2D Gaussians or confidence maps, as explicit input signals, training the model to respond directly to user intent.<n>We evaluate on seven datasets ranging from standard benchmarks to more challenging domains, including paintings, egocentric views, and X-ray data.
  arXiv  Detail & Related papers  (2026-06-21T20:30:41Z)
• Beyond Spatial Compression: Interface-Centric Generative States for Open-World 3D Structure [55.17771962316751]
  Current 3D tokenizers treat representation as spatial compression, but leave component ownership and attachment validity implicit.<n>We formulate an alternative view, interface-centric generative states, in which tokenization constructs an operational state rather than a passive compressed code.<n>We instantiate this formulation with Component-Conditioned Canonical Local Tokens (C2LT-3D), factorizing representation into canonical local geometry, partition-conditioned context, and relational seam variables.
  arXiv  Detail & Related papers  (2026-05-11T12:09:28Z)
• INSID3: Training-Free In-Context Segmentation with DINOv3 [63.961377087673476]
  INSID3 is a training-free approach that segments concepts at varying granularities only from frozen DINOv3 features.<n>It achieves state-of-the-art results across one-shot semantic, part, and personalized segmentation, outperforming previous work by +7.5 % mIoU.
  arXiv  Detail & Related papers  (2026-03-30T14:16:37Z)
• DynaPURLS: Dynamic Refinement of Part-aware Representations for Skeleton-based Zero-Shot Action Recognition [51.80782323686666]
  We introduce textbfDynaPURLS, a unified framework that establishes robust, multi-scale visual-semantic correspondences.<n>Our framework leverages a large language model to generate hierarchical textual descriptions that encompass both global movements and local body-part dynamics.<n>Experiments on three large-scale benchmark datasets, including NTU RGB+D 60/120 and PKU-MMD, demonstrate that DynaPURLS significantly outperforms prior art.
  arXiv  Detail & Related papers  (2025-12-12T10:39:10Z)
• MuSc-V2: Zero-Shot Multimodal Industrial Anomaly Classification and Segmentation with Mutual Scoring of Unlabeled Samples [8.323617762880533]
  We propose a Mutual Scoring framework (MuSc-V2) for zero-shot anomaly classification (AC/AS)<n>Our framework flexibly works on both the full dataset and smaller subsets with consistently robust performance.<n>In aid of the novel framework, MuSc-V2 achieves significant performance improvements.
  arXiv  Detail & Related papers  (2025-11-13T07:47:37Z)
• VesSAM: Efficient Multi-Prompting for Segmenting Complex Vessel [68.24765319399286]
  We present VesSAM, a powerful and efficient framework tailored for 2D vessel segmentation.<n>VesSAM integrates (1) a convolutional adapter to enhance local texture features, (2) a multi-prompt encoder that fuses anatomical prompts, and (3) a lightweight mask decoder to reduce jagged artifacts.<n>VesSAM consistently outperforms state-of-the-art PEFT-based SAM variants by over 10% Dice and 13% IoU.
  arXiv  Detail & Related papers  (2025-11-02T15:47:05Z)
• Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing [55.291219073365546]
  Open-Vocabulary Remote Sensing Image (OVRSIS) is an emerging task that adapts Open-Vocabulary (OVS) to the remote sensing (RS) domain.<n>textbfRSKT-Seg is a novel open-vocabulary segmentation framework tailored for remote sensing.<n> RSKT-Seg consistently outperforms strong OVS baselines by +3.8 mIoU and +5.9 mACC, while achieving 2x faster inference through efficient aggregation.
  arXiv  Detail & Related papers  (2025-09-15T15:24:49Z)
• GeoSAM2: Unleashing the Power of SAM2 for 3D Part Segmentation [81.0871900167463]
  We introduce GeoSAM2, a prompt-controllable framework for 3D part segmentation.<n>Given a textureless object, we render normal and point maps from predefined viewpoints.<n>We accept simple 2D prompts - clicks or boxes - to guide part selection.<n>The predicted masks are back-projected to the object and aggregated across views.
  arXiv  Detail & Related papers  (2025-08-19T17:58:51Z)
• DescriptorMedSAM: Language-Image Fusion with Multi-Aspect Text Guidance for Medical Image Segmentation [10.598528392988062]
  We propose DescriptorMedSAM, a lightweight extension of MedSAM that incorporates structured text prompts to enable click-free segmentation.<n>DescriptorMedSAM employs a CLIP text encoder to convert radiology-style descriptors into dense embeddings.<n>We demonstrate that structured language prompts can effectively replace spatial interactions, delivering strong zero-shot performance and rapid few-shot adaptation.
  arXiv  Detail & Related papers  (2025-03-18T01:35:34Z)
• S4M: Segment Anything with 4 Extreme Points [2.052287962818431]
  Segment Anything Model (SAM) primarily relies on sparse prompts such as point or bounding box.<n>S4M augments SAM by leveraging extreme points -- the top-, bottom-, left-, and right-most points of an instance -- prompts.<n>S4M outperforms other SAM-based approaches on three endoscopic surgical datasets.
  arXiv  Detail & Related papers  (2025-03-07T16:02:11Z)
• Interactive Segment Anything NeRF with Feature Imitation [20.972098365110426]
  We propose to imitate the backbone feature of off-the-shelf perception models to achieve zero-shot semantic segmentation with NeRF. Our framework reformulates the segmentation process by directly rendering semantic features and only applying the decoder from perception models. Furthermore, we can project the learned semantics onto extracted mesh surfaces for real-time interaction.
  arXiv  Detail & Related papers  (2023-05-25T16:44:51Z)
• A Simple Framework for Open-Vocabulary Segmentation and Detection [85.21641508535679]
  We present OpenSeeD, a simple Open-vocabulary and Detection framework that jointly learns from different segmentation and detection datasets. We first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. After pre-training, our model exhibits competitive or stronger zero-shot transferability for both segmentation and detection.
  arXiv  Detail & Related papers  (2023-03-14T17:58:34Z)
• Transforming the Interactive Segmentation for Medical Imaging [34.57242805353604]
  The goal of this paper is to interactively refine the automatic segmentation on challenging structures that fall behind human performance. We propose a novel Transformer-based architecture for Interactive (TIS) Our proposed architecture is composed of Transformer Decoder variants, which naturally fulfills feature comparison with the attention mechanisms.
  arXiv  Detail & Related papers  (2022-08-20T03:28:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.