SWAG: Long-term Surgical Workflow Prediction with Generative-based Anticipation

• URL: http://arxiv.org/abs/2412.18849v4
• Date: Sun, 15 Jun 2025 09:17:03 GMT
• Title: SWAG: Long-term Surgical Workflow Prediction with Generative-based Anticipation
• Abstract summary: We propose SWAG (Surgical Anticipative Generation), a framework that combines phase recognition and anticipation.<n>This paper investigates two distinct decoding methods - single-pass (SP) and auto-regressive (AR)<n>We propose a novel embedding approach using class transition probabilities to enhance the accuracy of phase anticipation.
• Score: 43.95663763660549
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While existing approaches excel at recognising current surgical phases, they provide limited foresight and intraoperative guidance into future procedural steps. Similarly, current anticipation methods are constrained to predicting short-term and single events, neglecting the dense, repetitive, and long sequential nature of surgical workflows. To address these needs and limitations, we propose SWAG (Surgical Workflow Anticipative Generation), a framework that combines phase recognition and anticipation using a generative approach. This paper investigates two distinct decoding methods - single-pass (SP) and auto-regressive (AR) - to generate sequences of future surgical phases at minute intervals over long horizons. We propose a novel embedding approach using class transition probabilities to enhance the accuracy of phase anticipation. Additionally, we propose a generative framework using remaining time regression to classification (R2C). SWAG was evaluated on two publicly available datasets, Cholec80 and AutoLaparo21. Our single-pass model with class transition probability embeddings (SP*) achieves 32.1% and 41.3% F1 scores over 20 and 30 minutes on Cholec80 and AutoLaparo21, respectively. Moreover, our approach competes with existing methods on phase remaining time regression, achieving weighted mean absolute errors of 0.32 and 0.48 minutes for 2- and 3-minute horizons. SWAG demonstrates versatility across generative decoding frame works and classification and regression tasks to create temporal continuity between surgical workflow recognition and anticipation. Our method provides steps towards intraoperative surgical workflow generation for anticipation. Project: https://maxboels.com/research/swag.

Related papers

• Steering the Flow: Inverting Face Recognition Models via Gradient-Guided Flow Matching [17.11318422163162]
  Model Inversion Attacks aim to reconstruct identities from face recognition models, exposing critical security vulnerabilities.<n>We propose a novel two-stage white-box model inversion method that reformulates inversion as a trajectory-steering task.<n>Under an identity-disjoint cross-evaluation setting, SFMI achieves an ACC of 0.9248, an FID of 22.61, and an LPIPS of 0.3874 on the ArcFace target.
  arXiv  Detail & Related papers  (2026-08-17T16:47:09Z)
• State-Change Learning for Prediction of Future Events in Endoscopic Videos [8.258852831142017]
  We introduce SurgFUTR, implementing this through a teacher-student architecture.<n>Video clips are compressed into state representations via Sinkhorn-Knopp clustering.<n>We establish SFPBench with five prediction tasks spanning short-term (triplets, events) and long-term (remaining surgery duration, phase and step transitions) horizons. Experiments across four datasets and three procedures show consistent improvements.
  arXiv  Detail & Related papers  (2025-10-14T18:25:03Z)
• Recognizing Surgical Phases Anywhere: Few-Shot Test-time Adaptation and Task-graph Guided Refinement [43.44675567476855]
  Surgical Phase Anywhere (SPA) is a lightweight framework for versatile surgical workflow understanding.<n>It adapts foundation models to institutional settings with minimal annotation.<n>It achieves state-of-the-art performance in few-shot surgical phase recognition across multiple institutions and procedures.
  arXiv  Detail & Related papers  (2025-06-25T08:56:13Z)
• CoStoDet-DDPM: Collaborative Training of Stochastic and Deterministic Models Improves Surgical Workflow Anticipation and Recognition [12.360775476995169]
  We introduce an innovative framework that incorporates inherent modeling through a denoising diffusion probabilistic model (DDPM) At the heart of our approach is a collaborative co-training paradigm. Experiments on the Cholec80 dataset show that for the anticipation task, our method achieves a 16% reduction in eMAE compared to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-03-13T09:59:05Z)
• Neural Finite-State Machines for Surgical Phase Recognition [30.912252237906724]
  Surgical phase recognition is crucial for applications in workflow optimization, performance evaluation, and real-time intervention guidance.<n>We propose the Neural Finite-State Machine (NFSM), a novel approach that enforces temporal coherence by integrating classical state-transition priors with modern neural networks.<n>We demonstrate state-of-the-art performance across multiple benchmarks, including a significant improvement on the BernBypass70 dataset.
  arXiv  Detail & Related papers  (2024-11-27T03:21:57Z)
• SuPRA: Surgical Phase Recognition and Anticipation for Intra-Operative Planning [46.57714869178571]
  We propose a dual approach that simultaneously recognises the current surgical phase and predicts upcoming ones. Our novel method, Surgical Phase Recognition and Anticipation (SuPRA), leverages past and current information for accurate intra-operative phase recognition.
  arXiv  Detail & Related papers  (2024-03-10T12:46:33Z)
• Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery [50.3022015601057]
  We propose a predictive neural network that is capable of understanding and predicting critical interactive aspects of surgical workflow from intra-abdominal video. We verify our approach on established surgical datasets and applications, including the detection and prediction of action triplets. Our results demonstrate the superiority of our approach compared to unstructured alternatives.
  arXiv  Detail & Related papers  (2024-02-03T00:58:05Z)
• GLSFormer : Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos [57.93194315839009]
  We propose a vision transformer-based approach to learn temporal features directly from sequence-level patches. We extensively evaluate our approach on two cataract surgery video datasets, Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-07-20T17:57:04Z)
• LoViT: Long Video Transformer for Surgical Phase Recognition [59.06812739441785]
  We present a two-stage method, called Long Video Transformer (LoViT) for fusing short- and long-term temporal information. Our approach outperforms state-of-the-art methods on the Cholec80 and AutoLaparo datasets consistently.
  arXiv  Detail & Related papers  (2023-05-15T20:06:14Z)
• DIR-AS: Decoupling Individual Identification and Temporal Reasoning for Action Segmentation [84.78383981697377]
  Fully supervised action segmentation works on frame-wise action recognition with dense annotations and often suffers from the over-segmentation issue. We develop a novel local-global attention mechanism with temporal pyramid dilation and temporal pyramid pooling for efficient multi-scale attention. We achieve state-of-the-art accuracy, eg, 82.8% (+2.6%) on GTEA and 74.7% (+1.2%) on Breakfast, which demonstrates the effectiveness of our proposed method.
  arXiv  Detail & Related papers  (2023-04-04T20:27:18Z)
• Post-Processing Temporal Action Detection [134.26292288193298]
  Temporal Action Detection (TAD) methods typically take a pre-processing step in converting an input varying-length video into a fixed-length snippet representation sequence. This pre-processing step would temporally downsample the video, reducing the inference resolution and hampering the detection performance in the original temporal resolution. We introduce a novel model-agnostic post-processing method without model redesign and retraining.
  arXiv  Detail & Related papers  (2022-11-27T19:50:37Z)
• Temporal Memory Relation Network for Workflow Recognition from Surgical Video [53.20825496640025]
  We propose a novel end-to-end temporal memory relation network (TMNet) for relating long-range and multi-scale temporal patterns. We have extensively validated our approach on two benchmark surgical video datasets.
  arXiv  Detail & Related papers  (2021-03-30T13:20:26Z)
• Temporal Context Aggregation Network for Temporal Action Proposal Refinement [93.03730692520999]
  Temporal action proposal generation is a challenging yet important task in the video understanding field. Current methods still suffer from inaccurate temporal boundaries and inferior confidence used for retrieval. We propose TCANet to generate high-quality action proposals through "local and global" temporal context aggregation.
  arXiv  Detail & Related papers  (2021-03-24T12:34:49Z)
• Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer [57.18185972461453]
  We introduce for the first time in surgical workflow analysis Transformer to reconsider the ignored complementary effects of spatial and temporal features for accurate phase recognition. Our framework is lightweight and processes the hybrid embeddings in parallel to achieve a high inference speed.
  arXiv  Detail & Related papers  (2021-03-17T15:12:55Z)
• Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures [13.338427746090796]
  We introduce two correlated surgical activities, phases and steps, for the laparoscopic gastric bypass procedure. We propose a Multi-task Multi-Stage Temporal Convolutional Network (MTMS-TCN) along with a multi-task Convolutional Neural Network (CNN) training setup. We evaluate the proposed method on a large video dataset consisting of 40 surgical procedures (Bypass40)
  arXiv  Detail & Related papers  (2021-02-24T11:18:19Z)
• TTPP: Temporal Transformer with Progressive Prediction for Efficient Action Anticipation [46.28067541184604]
  Video action anticipation aims to predict future action categories from observed frames. Current state-of-the-art approaches mainly resort to recurrent neural networks to encode history information into hidden states. This paper proposes a simple yet efficient Temporal Transformer with Progressive Prediction framework.
  arXiv  Detail & Related papers  (2020-03-07T07:59:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.