Temporal Propagation of Asymmetric Feature Pyramid for Surgical Scene Segmentation

• URL: http://arxiv.org/abs/2504.13440v1
• Date: Fri, 18 Apr 2025 03:41:23 GMT
• Title: Temporal Propagation of Asymmetric Feature Pyramid for Surgical Scene Segmentation
• Authors: Cheng Yuan, Yutong Ban,
• Abstract summary: Surgical scene segmentation is crucial for robot-assisted laparoscopic surgery understanding.<n>Current approaches face two challenges: (i) static image limitations and fine-grained structural details.<n>We present temporal asymmetric feature propagation network, a bidirectional attention architecture enabling cross-frame feature propagation.<n>Our framework uniquely enables both temporal guidance and contextual reasoning for surgical scene understanding.
• Score: 7.150163844454341
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Surgical scene segmentation is crucial for robot-assisted laparoscopic surgery understanding. Current approaches face two challenges: (i) static image limitations including ambiguous local feature similarities and fine-grained structural details, and (ii) dynamic video complexities arising from rapid instrument motion and persistent visual occlusions. While existing methods mainly focus on spatial feature extraction, they fundamentally overlook temporal dependencies in surgical video streams. To address this, we present temporal asymmetric feature propagation network, a bidirectional attention architecture enabling cross-frame feature propagation. The proposed method contains a temporal query propagator that integrates multi-directional consistency constraints to enhance frame-specific feature representation, and an aggregated asymmetric feature pyramid module that preserves discriminative features for anatomical structures and surgical instruments. Our framework uniquely enables both temporal guidance and contextual reasoning for surgical scene understanding. Comprehensive evaluations on two public benchmarks show the proposed method outperforms the current SOTA methods by a large margin, with +16.4\% mIoU on EndoVis2018 and +3.3\% mAP on Endoscapes2023. The code will be publicly available after paper acceptance.

Related papers

• Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition [64.56321246196859]
  We propose a novel dyNamically Evolving dUal skeleton-semantic syneRgistic framework.<n>We first construct the spatial-temporal evolving micro-prototypes and integrate dynamic context-aware side information.<n>We introduce the spatial compression and temporal memory mechanisms to guide the growth of spatial-temporal micro-prototypes.
  arXiv  Detail & Related papers  (2024-11-18T05:16:11Z)
• Surgical Scene Segmentation by Transformer With Asymmetric Feature Enhancement [7.150163844454341]
  Vision-specific transformer method is a promising way for surgical scene understanding. We propose a novel Transformer-based framework with an Asymmetric Feature Enhancement module (TAFE) The proposed method outperforms the SOTA methods in several different surgical segmentation tasks and additionally proves its ability of fine-grained structure recognition.
  arXiv  Detail & Related papers  (2024-10-23T07:58:47Z)
• LACOSTE: Exploiting stereo and temporal contexts for surgical instrument segmentation [14.152207010509763]
  We propose a novel LACOSTE model that exploits Location-Agnostic COntexts in Stereo and TEmporal images for improved surgical instrument segmentation. We extensively validate our approach on three public surgical video datasets.
  arXiv  Detail & Related papers  (2024-09-14T08:17:56Z)
• WeakSurg: Weakly supervised surgical instrument segmentation using temporal equivariance and semantic continuity [14.448593791011204]
  We propose a weakly supervised surgical instrument segmentation with only instrument presence labels. We take the inherent temporal attributes of surgical video into account and extend a two-stage weakly supervised segmentation paradigm. Experiments are validated on two surgical video datasets, including one cholecystectomy surgery benchmark and one real robotic left lateral segment liver surgery dataset.
  arXiv  Detail & Related papers  (2024-03-14T16:39:11Z)
• GLSFormer : Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos [57.93194315839009]
  We propose a vision transformer-based approach to learn temporal features directly from sequence-level patches. We extensively evaluate our approach on two cataract surgery video datasets, Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-07-20T17:57:04Z)
• LoViT: Long Video Transformer for Surgical Phase Recognition [59.06812739441785]
  We present a two-stage method, called Long Video Transformer (LoViT) for fusing short- and long-term temporal information. Our approach outperforms state-of-the-art methods on the Cholec80 and AutoLaparo datasets consistently.
  arXiv  Detail & Related papers  (2023-05-15T20:06:14Z)
• Spatiotemporal Multi-scale Bilateral Motion Network for Gait Recognition [3.1240043488226967]
  In this paper, motivated by optical flow, the bilateral motion-oriented features are proposed. We develop a set of multi-scale temporal representations that force the motion context to be richly described at various levels of temporal resolution.
  arXiv  Detail & Related papers  (2022-09-26T01:36:22Z)
• Efficient Global-Local Memory for Real-time Instrument Segmentation of Robotic Surgical Video [53.14186293442669]
  We identify two important clues for surgical instrument perception, including local temporal dependency from adjacent frames and global semantic correlation in long-range duration. We propose a novel dual-memory network (DMNet) to relate both global and local-temporal knowledge. Our method largely outperforms the state-of-the-art works on segmentation accuracy while maintaining a real-time speed.
  arXiv  Detail & Related papers  (2021-09-28T10:10:14Z)
• Improving Video Instance Segmentation via Temporal Pyramid Routing [61.10753640148878]
  Video Instance (VIS) is a new and inherently multi-task problem, which aims to detect, segment and track each instance in a video sequence. We propose a Temporal Pyramid Routing (TPR) strategy to conditionally align and conduct pixel-level aggregation from a feature pyramid pair of two adjacent frames. Our approach is a plug-and-play module and can be easily applied to existing instance segmentation methods.
  arXiv  Detail & Related papers  (2021-07-28T03:57:12Z)
• Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer [57.18185972461453]
  We introduce for the first time in surgical workflow analysis Transformer to reconsider the ignored complementary effects of spatial and temporal features for accurate phase recognition. Our framework is lightweight and processes the hybrid embeddings in parallel to achieve a high inference speed.
  arXiv  Detail & Related papers  (2021-03-17T15:12:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.