SurgiTrack: Fine-Grained Multi-Class Multi-Tool Tracking in Surgical Videos

• URL: http://arxiv.org/abs/2405.20333v1
• Date: Thu, 30 May 2024 17:59:10 GMT
• Title: SurgiTrack: Fine-Grained Multi-Class Multi-Tool Tracking in Surgical Videos
• Authors: Chinedu Innocent Nwoye, Nicolas Padoy,
• Abstract summary: This work sets a new standard in surgical tool tracking, providing dynamic trajectories for more adaptable and precise assistance in minimally invasive surgeries. We propose SurgiTrack, a novel deep learning method that leverages YOLOv7 for precise tool detection and employs an attention mechanism to model the originating direction of the tools, as a proxy to their operators, for tool re-identification.
• Score: 2.3810490830445183
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Accurate tool tracking is essential for the success of computer-assisted intervention. Previous efforts often modeled tool trajectories rigidly, overlooking the dynamic nature of surgical procedures, especially tracking scenarios like out-of-body and out-of-camera views. Addressing this limitation, the new CholecTrack20 dataset provides detailed labels that account for multiple tool trajectories in three perspectives: (1) intraoperative, (2) intracorporeal, and (3) visibility, representing the different types of temporal duration of tool tracks. These fine-grained labels enhance tracking flexibility but also increase the task complexity. Re-identifying tools after occlusion or re-insertion into the body remains challenging due to high visual similarity, especially among tools of the same category. This work recognizes the critical role of the tool operators in distinguishing tool track instances, especially those belonging to the same tool category. The operators' information are however not explicitly captured in surgical videos. We therefore propose SurgiTrack, a novel deep learning method that leverages YOLOv7 for precise tool detection and employs an attention mechanism to model the originating direction of the tools, as a proxy to their operators, for tool re-identification. To handle diverse tool trajectory perspectives, SurgiTrack employs a harmonizing bipartite matching graph, minimizing conflicts and ensuring accurate tool identity association. Experimental results on CholecTrack20 demonstrate SurgiTrack's effectiveness, outperforming baselines and state-of-the-art methods with real-time inference capability. This work sets a new standard in surgical tool tracking, providing dynamic trajectories for more adaptable and precise assistance in minimally invasive surgeries.

Related papers

• FeatureSORT: Essential Features for Effective Tracking [0.0]
  We introduce a novel tracker designed for online multiple object tracking with a focus on being simple, while being effective. By integrating distinct appearance features, including clothing color, style, and target direction, our tracker significantly enhances online tracking accuracy.
  arXiv  Detail & Related papers  (2024-07-05T04:37:39Z)
• Towards Completeness-Oriented Tool Retrieval for Large Language Models [60.733557487886635]
  Real-world systems often incorporate a wide array of tools, making it impractical to input all tools into Large Language Models. Existing tool retrieval methods primarily focus on semantic matching between user queries and tool descriptions. We propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT, which captures not only the semantic similarities between user queries and tool descriptions but also takes into account the collaborative information of tools.
  arXiv  Detail & Related papers  (2024-05-25T06:41:23Z)
• CholecTrack20: A Dataset for Multi-Class Multiple Tool Tracking in Laparoscopic Surgery [1.8076340162131013]
  CholecTrack20 is an extensive dataset meticulously annotated for multi-class multi-tool tracking across three perspectives. The dataset comprises 20 laparoscopic videos with over 35,000 frames and 65,000 annotated tool instances.
  arXiv  Detail & Related papers  (2023-12-12T15:18:15Z)
• Single-Shot and Multi-Shot Feature Learning for Multi-Object Tracking [55.13878429987136]
  We propose a simple yet effective two-stage feature learning paradigm to jointly learn single-shot and multi-shot features for different targets. Our method has achieved significant improvements on MOT17 and MOT20 datasets while reaching state-of-the-art performance on DanceTrack dataset.
  arXiv  Detail & Related papers  (2023-11-17T08:17:49Z)
• SAF-IS: a Spatial Annotation Free Framework for Instance Segmentation of Surgical Tools [10.295921059528636]
  We develop a framework for instance segmentation not relying on spatial annotations for training. Our solution only requires binary tool masks, obtainable using recent unsupervised approaches, and binary tool presence labels. We validate our framework on the EndoVis 2017 and 2018 segmentation datasets.
  arXiv  Detail & Related papers  (2023-09-04T17:13:06Z)
• Surgical tool classification and localization: results and methods from the MICCAI 2022 SurgToolLoc challenge [69.91670788430162]
  We present the results of the SurgLoc 2022 challenge. The goal was to leverage tool presence data as weak labels for machine learning models trained to detect tools. We conclude by discussing these results in the broader context of machine learning and surgical data science.
  arXiv  Detail & Related papers  (2023-05-11T21:44:39Z)
• Online Deep Clustering with Video Track Consistency [85.8868194550978]
  We propose an unsupervised clustering-based approach to learn visual features from video object tracks. We show that exploiting an unsupervised class-agnostic, yet noisy, track generator yields to better accuracy compared to relying on costly and precise track annotations.
  arXiv  Detail & Related papers  (2022-06-07T08:11:00Z)
• Correlation-Aware Deep Tracking [83.51092789908677]
  We propose a novel target-dependent feature network inspired by the self-/cross-attention scheme. Our network deeply embeds cross-image feature correlation in multiple layers of the feature network. Our model can be flexibly pre-trained on abundant unpaired images, leading to notably faster convergence than the existing methods.
  arXiv  Detail & Related papers  (2022-03-03T11:53:54Z)
• Crop-Transform-Paste: Self-Supervised Learning for Visual Tracking [137.26381337333552]
  In this work, we develop the Crop-Transform-Paste operation, which is able to synthesize sufficient training data. Since the object state is known in all synthesized data, existing deep trackers can be trained in routine ways without human annotation.
  arXiv  Detail & Related papers  (2021-06-21T07:40:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.