Efficient Frame Extraction: A Novel Approach Through Frame Similarity and Surgical Tool Tracking for Video Segmentation

• URL: http://arxiv.org/abs/2501.11153v1
• Date: Sun, 19 Jan 2025 19:36:09 GMT
• Title: Efficient Frame Extraction: A Novel Approach Through Frame Similarity and Surgical Tool Tracking for Video Segmentation
• Authors: Huu Phong Nguyen, Shekhar Madhav Khairnar, Sofia Garces Palacios, Amr Al-Abbas, Francisco Antunes, Bernardete Ribeiro, Melissa E. Hogg, Amer H. Zureikat, Patricio M. Polanco, Herbert Zeh III, Ganesh Sankaranarayanan,
• Abstract summary: We propose a technique that can efficiently eliminate redundant frames to reduce dataset size and computation time. Specifically, we compute the similarity between consecutive frames by tracking the movement of surgical tools. By adaptively selecting relevant frames, we achieve a tenfold reduction in the number of frames while improving accuracy by 4.32%.
• Score: 1.6092864505858449
• License:
• Abstract: The interest in leveraging Artificial Intelligence (AI) for surgical procedures to automate analysis has witnessed a significant surge in recent years. One of the primary tools for recording surgical procedures and conducting subsequent analyses, such as performance assessment, is through videos. However, these operative videos tend to be notably lengthy compared to other fields, spanning from thirty minutes to several hours, which poses a challenge for AI models to effectively learn from them. Despite this challenge, the foreseeable increase in the volume of such videos in the near future necessitates the development and implementation of innovative techniques to tackle this issue effectively. In this article, we propose a novel technique called Kinematics Adaptive Frame Recognition (KAFR) that can efficiently eliminate redundant frames to reduce dataset size and computation time while retaining useful frames to improve accuracy. Specifically, we compute the similarity between consecutive frames by tracking the movement of surgical tools. Our approach follows these steps: i) Tracking phase: a YOLOv8 model is utilized to detect tools presented in the scene, ii) Similarity phase: Similarities between consecutive frames are computed by estimating variation in the spatial positions and velocities of the tools, iii) Classification phase: A X3D CNN is trained to classify segmentation. We evaluate the effectiveness of our approach by analyzing datasets obtained through retrospective reviews of cases at two referral centers. The Gastrojejunostomy (GJ) dataset covers procedures performed between 2017 to 2021, while the Pancreaticojejunostomy (PJ) dataset spans from 2011 to 2022 at the same centers. By adaptively selecting relevant frames, we achieve a tenfold reduction in the number of frames while improving accuracy by 4.32% (from 0.749 to 0.7814).

Related papers

• Novel adaptation of video segmentation to 3D MRI: efficient zero-shot knee segmentation with SAM2 [1.6237741047782823]
  We introduce a method for zero-shot, single-prompt segmentation of 3D knee MRI by adapting Segment Anything Model 2. By treating slices from 3D medical volumes as individual video frames, we leverage SAM2's advanced capabilities to generate motion- and spatially-aware predictions. We demonstrate that SAM2 can efficiently perform segmentation tasks in a zero-shot manner with no additional training or fine-tuning.
  arXiv  Detail & Related papers  (2024-08-08T21:39:15Z)
• Self-Supervised Learning for Interventional Image Analytics: Towards Robust Device Trackers [6.262161803642583]
  We propose a novel approach to learn procedural features from a very large data cohort of over 16 million interventional X-ray frames. Our approach is based on a masked image modeling technique that leverages frame-based reconstruction to learn fine inter-frame temporal correspondences. Experiments show that our method achieves 66.31% reduction in maximum tracking error against reference solutions.
  arXiv  Detail & Related papers  (2024-05-02T10:18:22Z)
• Augmenting Efficient Real-time Surgical Instrument Segmentation in Video with Point Tracking and Segment Anything [9.338136334709818]
  We present a novel framework that combines an online point tracker with a lightweight SAM model that is fine-tuned for surgical instrument segmentation. Sparse points within the region of interest are tracked and used to prompt SAM throughout the video sequence, providing temporal consistency. Our method achieves promising performance that is comparable to XMem and transformer-based fully supervised segmentation methods.
  arXiv  Detail & Related papers  (2024-03-12T18:12:42Z)
• GLSFormer : Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos [57.93194315839009]
  We propose a vision transformer-based approach to learn temporal features directly from sequence-level patches. We extensively evaluate our approach on two cataract surgery video datasets, Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-07-20T17:57:04Z)
• LoViT: Long Video Transformer for Surgical Phase Recognition [59.06812739441785]
  We present a two-stage method, called Long Video Transformer (LoViT) for fusing short- and long-term temporal information. Our approach outperforms state-of-the-art methods on the Cholec80 and AutoLaparo datasets consistently.
  arXiv  Detail & Related papers  (2023-05-15T20:06:14Z)
• UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation [93.88170217725805]
  We propose a 3D medical image segmentation approach, named UNETR++, that offers both high-quality segmentation masks as well as efficiency in terms of parameters, compute cost, and inference speed. The core of our design is the introduction of a novel efficient paired attention (EPA) block that efficiently learns spatial and channel-wise discriminative features. Our evaluations on five benchmarks, Synapse, BTCV, ACDC, BRaTs, and Decathlon-Lung, reveal the effectiveness of our contributions in terms of both efficiency and accuracy.
  arXiv  Detail & Related papers  (2022-12-08T18:59:57Z)
• CaRTS: Causality-driven Robot Tool Segmentation from Vision and Kinematics Data [11.92904350972493]
  Vision-based segmentation of the robotic tool during robot-assisted surgery enables downstream applications, such as augmented reality feedback. With the introduction of deep learning, many methods were presented to solve instrument segmentation directly and solely from images. We present CaRTS, a causality-driven robot tool segmentation algorithm, that is designed based on a complementary causal model of the robot tool segmentation task.
  arXiv  Detail & Related papers  (2022-03-15T22:26:19Z)
• Efficient Global-Local Memory for Real-time Instrument Segmentation of Robotic Surgical Video [53.14186293442669]
  We identify two important clues for surgical instrument perception, including local temporal dependency from adjacent frames and global semantic correlation in long-range duration. We propose a novel dual-memory network (DMNet) to relate both global and local-temporal knowledge. Our method largely outperforms the state-of-the-art works on segmentation accuracy while maintaining a real-time speed.
  arXiv  Detail & Related papers  (2021-09-28T10:10:14Z)
• Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment [60.31418655784291]
  We propose a novel unsupervised domain adaptation framework which can simultaneously transfer multi-modality knowledge, i.e., both kinematic and visual data, from simulator to real robot. It remedies the domain gap with enhanced transferable features by using temporal cues in videos, and inherent correlations in multi-modal towards recognizing gesture. Results show that our approach recovers the performance with great improvement gains, up to 12.91% in ACC and 20.16% in F1score without using any annotations in real robot.
  arXiv  Detail & Related papers  (2021-03-06T09:10:03Z)
• Symmetric Dilated Convolution for Surgical Gesture Recognition [10.699258974625073]
  We propose a novel temporal convolutional architecture to automatically detect and segment surgical gestures. We devise our method with a symmetric dilation structure bridged by a self-attention module to encode and decode the long-term temporal patterns. We validate our approach on a fundamental robotic suturing task from the JIGSAWS dataset.
  arXiv  Detail & Related papers  (2020-07-13T13:34:48Z)
• Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video [64.44583693846751]
  We study the semi-supervised instrument segmentation from robotic surgical videos with sparse annotations. By exploiting generated data pairs, our framework can recover and even enhance temporal consistency of training sequences. Results show that our method outperforms the state-of-the-art semisupervised methods by a large margin.
  arXiv  Detail & Related papers  (2020-07-06T02:39:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.