Exploiting Segment-level Semantics for Online Phase Recognition from Surgical Videos

• URL: http://arxiv.org/abs/2111.11044v1
• Date: Mon, 22 Nov 2021 08:18:05 GMT
• Title: Exploiting Segment-level Semantics for Online Phase Recognition from Surgical Videos
• Abstract summary: We present a segment-attentive hierarchical consistency network (SAHC) for surgical phase recognition from videos. By regularizing the predictions of frames and their corresponding segments via a consistency loss, the network can generate semantic-consistent segments. Experimental results show that our method outperforms previous state-of-the-arts by a large margin.
• Score: 13.858624044986811
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic surgical phase recognition plays an important role in robot-assisted surgeries. Existing methods ignored a pivotal problem that surgical phases should be classified by learning segment-level semantics instead of solely relying on frame-wise information. In this paper, we present a segment-attentive hierarchical consistency network (SAHC) for surgical phase recognition from videos. The key idea is to extract hierarchical high-level semantic-consistent segments and use them to refine the erroneous predictions caused by ambiguous frames. To achieve it, we design a temporal hierarchical network to generate hierarchical high-level segments. Then, we introduce a hierarchical segment-frame attention (SFA) module to capture relations between the low-level frames and high-level segments. By regularizing the predictions of frames and their corresponding segments via a consistency loss, the network can generate semantic-consistent segments and then rectify the misclassified predictions caused by ambiguous low-level frames. We validate SAHC on two public surgical video datasets, i.e., the M2CAI16 challenge dataset and the Cholec80 dataset. Experimental results show that our method outperforms previous state-of-the-arts by a large margin, notably reaches 4.1% improvements on M2CAI16. Code will be released at GitHub upon acceptance.

Related papers

• ProBAG: Prototype-Guided Boundary-Aware Graph Diffusion for Weakly Supervised Histopathology Segmentation [1.8629748140948348]
  Weakly supervised semantic segmentation enables histopathology tissue segmentation from image-level annotations.<n>We propose ProBAG, a stage-1 pseudo-mask generator that combines dataset-specific visual prototypes with pathology-aligned CONCH text prototypes.<n> Experiments on BCSS-WSSS and LUAD-HistoSeg show consistent gains over recent WSSS approaches.
  arXiv  Detail & Related papers  (2026-08-12T08:05:07Z)
• Bi-Level Collaborative Learning for Few-Shot Scribble-Supervised Medical Image Segmentation [29.491024029816334]
  We propose a bi-level collaborative learning framework for scribble-supervised medical image segmentation.<n>An upper-level learnable superpixel model is introduced to provide region-structural priors for lower-level segmentation.<n>The proposed framework significantly outperforms existing state-of-the-art scribble-supervised methods on the ACDC and Prostate datasets.
  arXiv  Detail & Related papers  (2026-07-28T08:26:40Z)
• CARDIAG: A Dense Segment Classification Benchmark of Deep Learning Architectures for Coronary Angiography [0.0]
  We demonstrate a new benchmark for the assessment of deep learning models which densely classify pixels of coronary angiograms to one of SYNTAX classes (or background)<n>The evaluation covers 24 distinct architectures starting with classic convnets to recent state-space-based vision algorithms.
  arXiv  Detail & Related papers  (2026-07-24T09:35:18Z)
• HTT-Net: Hierarchical Text-guided Transition Modeling for Surgical Video Phase Recognition [22.19338002382572]
  We propose HTT-Net, a Hierarchical Text-guided Transition modeling Network for surgical video phase recognition.<n>Key idea is to introduce structured surgical semantic knowledge into phase-aware segment construction and semantic refinement.<n>Experiments on Cholec80 and LCRS-100 demonstrate the effectiveness of HTT-Net for robust surgical video phase recognition.
  arXiv  Detail & Related papers  (2026-07-18T11:49:40Z)
• ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking [15.83425997240828]
  ReSurgSAM2 is a two-stage referring surgical segmentation framework.<n>It uses cross-modal spatial-temporal Mamba to generate precise detection and segmentation results.<n>It incorporates a diversity-driven memory mechanism that maintains a credible and diverse memory bank, ensuring consistent long-term tracking.
  arXiv  Detail & Related papers  (2025-05-13T13:56:10Z)
• SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection [16.89965584177711]
  Recent open-vocabulary human-object interaction (OV-HOI) detection methods rely on large language model (LLM) for generating auxiliary descriptions and leverage knowledge distilled from CLIP to detect unseen interaction categories. Despite their effectiveness, these methods face two challenges: (1) feature granularity deficiency, due to reliance on last layer visual features for text alignment, leading to the neglect of crucial object-level details from intermediate layers; (2) semantic similarity confusion, resulting from CLIP's inherent biases toward certain classes, while LLM-generated descriptions based solely on labels fail to adequately capture inter-class similarities.
  arXiv  Detail & Related papers  (2025-03-01T09:26:05Z)
• Hierarchical Action Recognition: A Contrastive Video-Language Approach with Hierarchical Interactions [19.741453194665276]
  We formalize the novel task of hierarchical video recognition, and propose a video-language learning framework tailored for hierarchical recognition. Specifically, our framework encodes dependencies between hierarchical category levels, and applies a top-down constraint to filter recognition predictions. We demonstrate the efficacy of our approach for hierarchical recognition, significantly outperforming conventional methods.
  arXiv  Detail & Related papers  (2024-05-28T01:17:22Z)
• SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation [91.91385816767057]
  Open-vocabulary semantic segmentation strives to distinguish pixels into different semantic groups from an open set of categories. We propose a simple encoder-decoder, named SED, for open-vocabulary semantic segmentation. Our SED method achieves mIoU score of 31.6% on ADE20K with 150 categories at 82 millisecond ($ms$) per image on a single A6000.
  arXiv  Detail & Related papers  (2023-11-27T05:00:38Z)
• Feature Enhancer Segmentation Network (FES-Net) for Vessel Segmentation [19.455350961592742]
  We propose a novel feature enhancement segmentation network (FES-Net) that achieves accurate pixel-wise segmentation without requiring additional image enhancement steps. FES-Net directly processes the input image and utilizes four prompt convolutional blocks (PCBs) during downsampling. We evaluate the performance of FES-Net on four publicly available state-of-the-art datasets: DRIVE, STARE, CHASE, and HRF.
  arXiv  Detail & Related papers  (2023-09-07T07:46:46Z)
• Self-Supervised Correction Learning for Semi-Supervised Biomedical Image Segmentation [84.58210297703714]
  We propose a self-supervised correction learning paradigm for semi-supervised biomedical image segmentation. We design a dual-task network, including a shared encoder and two independent decoders for segmentation and lesion region inpainting. Experiments on three medical image segmentation datasets for different tasks demonstrate the outstanding performance of our method.
  arXiv  Detail & Related papers  (2023-01-12T08:19:46Z)
• Reliable Joint Segmentation of Retinal Edema Lesions in OCT Images [55.83984261827332]
  In this paper, we propose a novel reliable multi-scale wavelet-enhanced transformer network. We develop a novel segmentation backbone that integrates a wavelet-enhanced feature extractor network and a multi-scale transformer module. Our proposed method achieves better segmentation accuracy with a high degree of reliability as compared to other state-of-the-art segmentation approaches.
  arXiv  Detail & Related papers  (2022-12-01T07:32:56Z)
• Retrieval of surgical phase transitions using reinforcement learning [11.130363429095048]
  We introduce a novel reinforcement learning formulation for offline phase transition retrieval. By construction, our model does not produce spurious and noisy phase transitions, but contiguous phase blocks. We compare our method against the recent top-performing frame-based approaches TeCNO and Trans-SVNet.
  arXiv  Detail & Related papers  (2022-08-01T14:43:15Z)
• Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation [58.74791043631219]
  We propose a novel framework STswinCL that explores the complementary intra- and inter-video relations to boost segmentation performance. We extensively validate our approach on two public surgical video benchmarks, including EndoVis18 Challenge and CaDIS dataset. Experimental results demonstrate the promising performance of our method, which consistently exceeds previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-29T05:52:23Z)
• Deep Hierarchical Semantic Segmentation [76.40565872257709]
  hierarchical semantic segmentation (HSS) aims at structured, pixel-wise description of visual observation in terms of a class hierarchy. HSSN casts HSS as a pixel-wise multi-label classification task, only bringing minimal architecture change to current segmentation models. With hierarchy-induced margin constraints, HSSN reshapes the pixel embedding space, so as to generate well-structured pixel representations.
  arXiv  Detail & Related papers  (2022-03-27T15:47:44Z)
• TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery [60.439434751619736]
  We propose TraSeTR, a Track-to-Segment Transformer that exploits tracking cues to assist surgical instrument segmentation. TraSeTR jointly reasons about the instrument type, location, and identity with instance-level predictions. The effectiveness of our method is demonstrated with state-of-the-art instrument type segmentation results on three public datasets.
  arXiv  Detail & Related papers  (2022-02-17T05:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.