TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery

• URL: http://arxiv.org/abs/2202.08453v1
• Date: Thu, 17 Feb 2022 05:52:18 GMT
• Title: TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery
• Authors: Zixu Zhao, Yueming Jin, Pheng-Ann Heng
• Abstract summary: We propose TraSeTR, a Track-to-Segment Transformer that exploits tracking cues to assist surgical instrument segmentation. TraSeTR jointly reasons about the instrument type, location, and identity with instance-level predictions. The effectiveness of our method is demonstrated with state-of-the-art instrument type segmentation results on three public datasets.
• Score: 60.439434751619736
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Surgical instrument segmentation -- in general a pixel classification task -- is fundamentally crucial for promoting cognitive intelligence in robot-assisted surgery (RAS). However, previous methods are struggling with discriminating instrument types and instances. To address the above issues, we explore a mask classification paradigm that produces per-segment predictions. We propose TraSeTR, a novel Track-to-Segment Transformer that wisely exploits tracking cues to assist surgical instrument segmentation. TraSeTR jointly reasons about the instrument type, location, and identity with instance-level predictions i.e., a set of class-bbox-mask pairs, by decoding query embeddings. Specifically, we introduce the prior query that encoded with previous temporal knowledge, to transfer tracking signals to current instances via identity matching. A contrastive query learning strategy is further applied to reshape the query feature space, which greatly alleviates the tracking difficulty caused by large temporal variations. The effectiveness of our method is demonstrated with state-of-the-art instrument type segmentation results on three public datasets, including two RAS benchmarks from EndoVis Challenges and one cataract surgery dataset CaDIS.

Related papers

• SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge [72.97934765570069]
  We release the first multimodal, publicly available, in-vivo, dataset for surgical action recognition and semantic instrumentation segmentation, containing 50 suturing video segments of Robotic Assisted Radical Prostatectomy (RARP) The aim of the challenge is to enable researchers to leverage the scale of the provided dataset and develop robust and highly accurate single-task action recognition and tool segmentation approaches in the surgical domain. A total of 12 teams participated in the challenge, contributing 7 action recognition methods, 9 instrument segmentation techniques, and 4 multitask approaches that integrated both action recognition and instrument segmentation.
  arXiv  Detail & Related papers  (2023-12-31T13:32:18Z)
• PWISeg: Point-based Weakly-supervised Instance Segmentation for Surgical Instruments [27.89003436883652]
  We propose a weakly-supervised surgical instrument segmentation approach, named Point-based Weakly-supervised Instance (PWISeg) PWISeg adopts an FCN-based architecture with point-to-box and point-to-mask branches to model the relationships between feature points and bounding boxes. Based on this, we propose a key pixel association loss and a key pixel distribution loss, driving the point-to-mask branch to generate more accurate segmentation predictions.
  arXiv  Detail & Related papers  (2023-11-16T11:48:29Z)
• SegMatch: A semi-supervised learning method for surgical instrument segmentation [10.223709180135419]
  We propose SegMatch, a semi supervised learning method to reduce the need for expensive annotation for laparoscopic and robotic surgical images. SegMatch builds on FixMatch, a widespread semi supervised classification pipeline combining consistency regularization and pseudo labelling. Our results demonstrate that adding unlabelled data for training purposes allows us to surpass the performance of fully supervised approaches.
  arXiv  Detail & Related papers  (2023-08-09T21:30:18Z)
• Transforming the Interactive Segmentation for Medical Imaging [34.57242805353604]
  The goal of this paper is to interactively refine the automatic segmentation on challenging structures that fall behind human performance. We propose a novel Transformer-based architecture for Interactive (TIS) Our proposed architecture is composed of Transformer Decoder variants, which naturally fulfills feature comparison with the attention mechanisms.
  arXiv  Detail & Related papers  (2022-08-20T03:28:23Z)
• SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation [111.61261419566908]
  Deep neural networks (DNNs) are usually trained on a closed set of semantic classes. They are ill-equipped to handle previously-unseen objects. detecting and localizing such objects is crucial for safety-critical applications such as perception for automated driving.
  arXiv  Detail & Related papers  (2021-04-30T07:58:19Z)
• Co-Generation and Segmentation for Generalized Surgical Instrument Segmentation on Unlabelled Data [49.419268399590045]
  Surgical instrument segmentation for robot-assisted surgery is needed for accurate instrument tracking and augmented reality overlays. Deep learning-based methods have shown state-of-the-art performance for surgical instrument segmentation, but their results depend on labelled data. In this paper, we demonstrate the limited generalizability of these methods on different datasets, including human robot-assisted surgeries.
  arXiv  Detail & Related papers  (2021-03-16T18:41:18Z)
• NINEPINS: Nuclei Instance Segmentation with Point Annotations [2.19221864553448]
  We propose an algorithm for instance segmentation that uses pseudo-label segmentations generated automatically from point annotations. With the generated segmentation masks, the proposed method trains a modified version of HoVer-Net model to achieve instance segmentation. Experimental results show that the proposed method is robust to inaccuracies in point annotations and comparison with Hover-Net trained with fully annotated instance masks shows that a degradation in segmentation performance does not always imply a degradation in higher order tasks such as tissue classification.
  arXiv  Detail & Related papers  (2020-06-24T08:28:52Z)
• Robust Medical Instrument Segmentation Challenge 2019 [56.148440125599905]
  Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. Our challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap.
  arXiv  Detail & Related papers  (2020-03-23T14:35:08Z)
• VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images [121.31355003451152]
  Large Scale Vertebrae Challenge (VerSe) was organised in conjunction with the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) in 2019 and 2020. We present the the results of this evaluation and further investigate the performance-variation at vertebra-level, scan-level, and at different fields-of-view.
  arXiv  Detail & Related papers  (2020-01-24T21:09:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.