MuST: Multi-Scale Transformers for Surgical Phase Recognition

• URL: http://arxiv.org/abs/2407.17361v1
• Date: Wed, 24 Jul 2024 15:38:20 GMT
• Title: MuST: Multi-Scale Transformers for Surgical Phase Recognition
• Authors: Alejandra Pérez, Santiago Rodríguez, Nicolás Ayobi, Nicolás Aparicio, Eugénie Dessevres, Pablo Arbeláez,
• Abstract summary: Phase recognition in surgical videos is crucial for enhancing computer-aided surgical systems. Existing methods often rely on fixed temporal windows for video analysis to identify dynamic surgical phases. We propose Multi-Scale Transformers for Surgical Phase Recognition (MuST), a novel Transformer-based approach.
• Score: 40.047145788604716
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Phase recognition in surgical videos is crucial for enhancing computer-aided surgical systems as it enables automated understanding of sequential procedural stages. Existing methods often rely on fixed temporal windows for video analysis to identify dynamic surgical phases. Thus, they struggle to simultaneously capture short-, mid-, and long-term information necessary to fully understand complex surgical procedures. To address these issues, we propose Multi-Scale Transformers for Surgical Phase Recognition (MuST), a novel Transformer-based approach that combines a Multi-Term Frame encoder with a Temporal Consistency Module to capture information across multiple temporal scales of a surgical video. Our Multi-Term Frame Encoder computes interdependencies across a hierarchy of temporal scales by sampling sequences at increasing strides around the frame of interest. Furthermore, we employ a long-term Transformer encoder over the frame embeddings to further enhance long-term reasoning. MuST achieves higher performance than previous state-of-the-art methods on three different public benchmarks.

Related papers

• Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition [2.10407185597278]
  We propose the Multi-Scale Action Transformer (MS-AST) for offline surgical phase recognition and the Multi-Scale Action Causal Transformer (MS-ASCT) for online surgical phase recognition. Our method can achieve 95.26% and 96.15% accuracy on the Cholec80 dataset for online and offline surgical phase recognition, respectively.
  arXiv  Detail & Related papers  (2024-01-22T01:34:03Z)
• SurgPLAN: Surgical Phase Localization Network for Phase Recognition [14.857715124466594]
  We propose a Surgical Phase LocAlization Network, named SurgPLAN, to facilitate a more accurate and stable surgical phase recognition. We first devise a Pyramid SlowFast (PSF) architecture to serve as the visual backbone to capture multi-scale spatial and temporal features by two branches with different frame sampling rates.
  arXiv  Detail & Related papers  (2023-11-16T15:39:01Z)
• Ring Attention with Blockwise Transformers for Near-Infinite Context [88.61687950039662]
  We present a novel approach, Ring Attention with Blockwise Transformers (Ring Attention), which leverages blockwise computation of self-attention and feedforward to distribute long sequences across multiple devices. Our approach enables training and inference of sequences that are up to device count times longer than those achievable by prior memory-efficient Transformers.
  arXiv  Detail & Related papers  (2023-10-03T08:44:50Z)
• GLSFormer : Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos [57.93194315839009]
  We propose a vision transformer-based approach to learn temporal features directly from sequence-level patches. We extensively evaluate our approach on two cataract surgery video datasets, Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-07-20T17:57:04Z)
• LoViT: Long Video Transformer for Surgical Phase Recognition [59.06812739441785]
  We present a two-stage method, called Long Video Transformer (LoViT) for fusing short- and long-term temporal information. Our approach outperforms state-of-the-art methods on the Cholec80 and AutoLaparo datasets consistently.
  arXiv  Detail & Related papers  (2023-05-15T20:06:14Z)
• FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification [53.55504611255664]
  FormerTime is a hierarchical representation model for improving the classification capacity for the multivariate time series classification task. It exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism.
  arXiv  Detail & Related papers  (2023-02-20T07:46:14Z)
• Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer [57.18185972461453]
  We introduce for the first time in surgical workflow analysis Transformer to reconsider the ignored complementary effects of spatial and temporal features for accurate phase recognition. Our framework is lightweight and processes the hybrid embeddings in parallel to achieve a high inference speed.
  arXiv  Detail & Related papers  (2021-03-17T15:12:55Z)
• Multi-frame Feature Aggregation for Real-time Instrument Segmentation in Endoscopic Video [11.100734994959419]
  We propose a novel Multi-frame Feature Aggregation (MFFA) module to aggregate video frame features temporally and spatially. We also develop a method that can randomly synthesize a surgical frame sequence from a single labeled frame to assist network training.
  arXiv  Detail & Related papers  (2020-11-17T16:27:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.