Less is More? Revisiting the Importance of Frame Rate in Real-Time Zero-Shot Surgical Video Segmentation

• URL: http://arxiv.org/abs/2502.20934v2
• Date: Mon, 07 Apr 2025 13:22:10 GMT
• Title: Less is More? Revisiting the Importance of Frame Rate in Real-Time Zero-Shot Surgical Video Segmentation
• Authors: Utku Ozbulak, Seyed Amir Mousavi, Francesca Tozzi, Niki Rashidian, Wouter Willaert, Wesley De Neve, Joris Vankerschaver,
• Abstract summary: We investigate the impact of frame rate on zero-shot surgical video segmentation, evaluating SAM2's effectiveness across multiple frame sampling rates for cholecystectomy procedures.<n>Surprisingly, our findings indicate that in conventional evaluation settings, frame rates as low as a single frame per second can outperform 25 FPS, as fewer frames smooth out segmentation inconsistencies.<n>In a real-time streaming scenario, higher frame rates yield superior temporal coherence and stability, particularly for dynamic objects such as surgical graspers.
• Score: 1.0536099636804035
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-time video segmentation is a promising feature for AI-assisted surgery, providing intraoperative guidance by identifying surgical tools and anatomical structures. However, deploying state-of-the-art segmentation models, such as SAM2, in real-time settings is computationally demanding, which makes it essential to balance frame rate and segmentation performance. In this study, we investigate the impact of frame rate on zero-shot surgical video segmentation, evaluating SAM2's effectiveness across multiple frame sampling rates for cholecystectomy procedures. Surprisingly, our findings indicate that in conventional evaluation settings, frame rates as low as a single frame per second can outperform 25 FPS, as fewer frames smooth out segmentation inconsistencies. However, when assessed in a real-time streaming scenario, higher frame rates yield superior temporal coherence and stability, particularly for dynamic objects such as surgical graspers. Finally, we investigate human perception of real-time surgical video segmentation among professionals who work closely with such data and find that respondents consistently prefer high FPS segmentation mask overlays, reinforcing the importance of real-time evaluation in AI-assisted surgery.

Related papers

• Surgeons vs. Computer Vision: A comparative analysis on surgical phase recognition capabilities [65.66373425605278]
  Automated Surgical Phase Recognition (SPR) uses Artificial Intelligence (AI) to segment the surgical workflow into its key events. Previous research has focused on short and linear surgical procedures and has not explored if temporal context influences experts' ability to better classify surgical phases. This research addresses these gaps, focusing on Robot-Assisted Partial Nephrectomy (RAPN) as a highly non-linear procedure.
  arXiv  Detail & Related papers  (2025-04-26T15:37:22Z)
• Efficient Frame Extraction: A Novel Approach Through Frame Similarity and Surgical Tool Tracking for Video Segmentation [1.6092864505858449]
  We propose a technique that can efficiently eliminate redundant frames to reduce dataset size and computation time.<n>Specifically, we compute the similarity between consecutive frames by tracking the movement of surgical tools.<n>By adaptively selecting relevant frames, we achieve a tenfold reduction in the number of frames while improving accuracy by 4.32%.
  arXiv  Detail & Related papers  (2025-01-19T19:36:09Z)
• WeakSurg: Weakly supervised surgical instrument segmentation using temporal equivariance and semantic continuity [14.448593791011204]
  We propose a weakly supervised surgical instrument segmentation with only instrument presence labels. We take the inherent temporal attributes of surgical video into account and extend a two-stage weakly supervised segmentation paradigm. Experiments are validated on two surgical video datasets, including one cholecystectomy surgery benchmark and one real robotic left lateral segment liver surgery dataset.
  arXiv  Detail & Related papers  (2024-03-14T16:39:11Z)
• Augmenting Efficient Real-time Surgical Instrument Segmentation in Video with Point Tracking and Segment Anything [9.338136334709818]
  We present a novel framework that combines an online point tracker with a lightweight SAM model that is fine-tuned for surgical instrument segmentation. Sparse points within the region of interest are tracked and used to prompt SAM throughout the video sequence, providing temporal consistency. Our method achieves promising performance that is comparable to XMem and transformer-based fully supervised segmentation methods.
  arXiv  Detail & Related papers  (2024-03-12T18:12:42Z)
• GLSFormer : Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos [57.93194315839009]
  We propose a vision transformer-based approach to learn temporal features directly from sequence-level patches. We extensively evaluate our approach on two cataract surgery video datasets, Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-07-20T17:57:04Z)
• A spatio-temporal network for video semantic segmentation in surgical videos [11.548181453080087]
  We propose a novel architecture for modelling temporal relationships in videos. The proposed model includes a decoder to enable semantic video segmentation. The proposed decoder can be used on top of any segmentation encoder to improve temporal consistency.
  arXiv  Detail & Related papers  (2023-06-19T16:36:48Z)
• LoViT: Long Video Transformer for Surgical Phase Recognition [59.06812739441785]
  We present a two-stage method, called Long Video Transformer (LoViT) for fusing short- and long-term temporal information. Our approach outperforms state-of-the-art methods on the Cholec80 and AutoLaparo datasets consistently.
  arXiv  Detail & Related papers  (2023-05-15T20:06:14Z)
• Distortion-Aware Network Pruning and Feature Reuse for Real-time Video Segmentation [49.17930380106643]
  We propose a novel framework to speed up any architecture with skip-connections for real-time vision tasks. Specifically, at the arrival of each frame, we transform the features from the previous frame to reuse them at specific spatial bins. We then perform partial computation of the backbone network on the regions of the current frame that captures temporal differences between the current and previous frame.
  arXiv  Detail & Related papers  (2022-06-20T07:20:02Z)
• Efficient Global-Local Memory for Real-time Instrument Segmentation of Robotic Surgical Video [53.14186293442669]
  We identify two important clues for surgical instrument perception, including local temporal dependency from adjacent frames and global semantic correlation in long-range duration. We propose a novel dual-memory network (DMNet) to relate both global and local-temporal knowledge. Our method largely outperforms the state-of-the-art works on segmentation accuracy while maintaining a real-time speed.
  arXiv  Detail & Related papers  (2021-09-28T10:10:14Z)
• Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video [11.100734994959419]
  We propose a novel Multi-frame Feature Aggregation (MFFA) module to aggregate video frame features temporally and spatially. We also develop a method that can randomly synthesize a surgical frame sequence from a single labeled frame to assist network training.
  arXiv  Detail & Related papers  (2020-11-17T16:27:27Z)
• Efficient Semantic Video Segmentation with Per-frame Inference [117.97423110566963]
  In this work, we process efficient semantic video segmentation in a per-frame fashion during the inference process. We employ compact models for real-time execution. To narrow the performance gap between compact models and large models, new knowledge distillation methods are designed.
  arXiv  Detail & Related papers  (2020-02-26T12:24:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.