ReSW-VL: Representation Learning for Surgical Workflow Analysis Using Vision-Language Model

• URL: http://arxiv.org/abs/2505.13746v1
• Date: Mon, 19 May 2025 21:44:37 GMT
• Title: ReSW-VL: Representation Learning for Surgical Workflow Analysis Using Vision-Language Model
• Authors: Satoshi Kondo,
• Abstract summary: Surgical phase recognition from video is a technology that automatically classifies the progress of a surgical procedure.<n>Recent advances in surgical phase recognition technology have focused primarily on Transform-based methods.<n>We propose a method for representation learning in surgical workflow analysis using a vision-language model.
• Score: 0.07143413923310668
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Surgical phase recognition from video is a technology that automatically classifies the progress of a surgical procedure and has a wide range of potential applications, including real-time surgical support, optimization of medical resources, training and skill assessment, and safety improvement. Recent advances in surgical phase recognition technology have focused primarily on Transform-based methods, although methods that extract spatial features from individual frames using a CNN and video features from the resulting time series of spatial features using time series modeling have shown high performance. However, there remains a paucity of research on training methods for CNNs employed for feature extraction or representation learning in surgical phase recognition. In this study, we propose a method for representation learning in surgical workflow analysis using a vision-language model (ReSW-VL). Our proposed method involves fine-tuning the image encoder of a CLIP (Convolutional Language Image Model) vision-language model using prompt learning for surgical phase recognition. The experimental results on three surgical phase recognition datasets demonstrate the effectiveness of the proposed method in comparison to conventional methods.

Related papers

• Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation [51.222684687924215]
  Surgical video-language pretraining faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data.<n>We propose a hierarchical knowledge augmentation approach and a novel Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining framework to tackle these issues.
  arXiv  Detail & Related papers  (2024-09-30T22:21:05Z)
• SurgPETL: Parameter-Efficient Image-to-Surgical-Video Transfer Learning for Surgical Phase Recognition [9.675072799670458]
  "Image pre-training followed by video fine-tuning" for high-dimensional video data poses significant performance bottlenecks. In this paper, we develop a parameter-efficient transfer learning benchmark SurgPETL for surgical phase recognition. We conduct extensive experiments with three advanced methods based on ViTs of two distinct scales pre-trained on five large-scale natural and medical datasets.
  arXiv  Detail & Related papers  (2024-09-30T08:33:50Z)
• Thoracic Surgery Video Analysis for Surgical Phase Recognition [0.08706730566331035]
  We analyse and evaluate both frame-based and video clipping-based phase recognition on thoracic surgery dataset consisting of 11 classes of phases. We show that Masked Video Distillation(MVD) exhibits superior performance, achieving a top-1 accuracy of 72.9%, compared to 52.31% achieved by ImageNet ViT.
  arXiv  Detail & Related papers  (2024-06-13T14:47:57Z)
• Interactive Generation of Laparoscopic Videos with Diffusion Models [1.5488613349551188]
  We show how to generate realistic laparoscopic images and videos by specifying a surgical action through text. We demonstrate the performance of our approach using the publicly available Cholec dataset family. We achieve an FID of 38.097 and an F1-score of 0.71.
  arXiv  Detail & Related papers  (2024-04-23T12:36:07Z)
• Efficient Surgical Tool Recognition via HMM-Stabilized Deep Learning [25.146476653453227]
  We propose an HMM-stabilized deep learning method for tool presence detection. A range of experiments confirm that the proposed approaches achieve better performance with lower training and running costs. These results suggest that popular deep learning approaches with over-complicated model structures may suffer from inefficient utilization of data.
  arXiv  Detail & Related papers  (2024-04-07T15:27:35Z)
• Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures [50.09187683845788]
  Recent advancements in surgical computer vision applications have been driven by vision-only models.<n>These methods rely on manually annotated surgical videos to predict a fixed set of object categories.<n>In this work, we put forward the idea that the surgical video lectures available through open surgical e-learning platforms can provide effective vision and language supervisory signals.
  arXiv  Detail & Related papers  (2023-07-27T22:38:12Z)
• GLSFormer : Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos [57.93194315839009]
  We propose a vision transformer-based approach to learn temporal features directly from sequence-level patches. We extensively evaluate our approach on two cataract surgery video datasets, Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-07-20T17:57:04Z)
• Dissecting Self-Supervised Learning Methods for Surgical Computer Vision [51.370873913181605]
  Self-Supervised Learning (SSL) methods have begun to gain traction in the general computer vision community. The effectiveness of SSL methods in more complex and impactful domains, such as medicine and surgery, remains limited and unexplored. We present an extensive analysis of the performance of these methods on the Cholec80 dataset for two fundamental and popular tasks in surgical context understanding, phase recognition and tool presence detection.
  arXiv  Detail & Related papers  (2022-07-01T14:17:11Z)
• Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures [70.69948035469467]
  We take advantage of the latest computer vision methodologies for generating 3D graphs from camera views. We then introduce the Multimodal Semantic Graph Scene (MSSG) which aims at providing unified symbolic and semantic representation of surgical procedures.
  arXiv  Detail & Related papers  (2021-06-09T14:35:44Z)
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery [84.73764603474413]
  We propose a novel online approach of multi-modal graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset.
  arXiv  Detail & Related papers  (2020-11-03T11:00:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.