Zero-shot Prompt-based Video Encoder for Surgical Gesture Recognition

• URL: http://arxiv.org/abs/2403.19786v2
• Date: Wed, 21 Aug 2024 19:54:02 GMT
• Title: Zero-shot Prompt-based Video Encoder for Surgical Gesture Recognition
• Authors: Mingxing Rao, Yinhong Qin, Soheil Kolouri, Jie Ying Wu, Daniel Moyer,
• Abstract summary: We develop a pre-trained vision-text model (CLIP) for gesture recognition in surgical videos. This can utilize extensive outside video data such as text, but also make use of label meta-data and weakly supervised contrastive losses. Experiments show that prompt-based video encoder outperforms standard encoders in surgical gesture recognition tasks.
• Score: 9.426097444566704
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Purpose: In order to produce a surgical gesture recognition system that can support a wide variety of procedures, either a very large annotated dataset must be acquired, or fitted models must generalize to new labels (so called "zero-shot" capability). In this paper we investigate the feasibility of latter option. Methods: Leveraging the Bridge-Prompt framework, we prompt-tune a pre-trained vision-text model (CLIP) for gesture recognition in surgical videos. This can utilize extensive outside video data such as text, but also make use of label meta-data and weakly supervised contrastive losses. Results: Our experiments show that prompt-based video encoder outperforms standard encoders in surgical gesture recognition tasks. Notably, it displays strong performance in zero-shot scenarios, where gestures/tasks that were not provided during the encoder training phase are included in the prediction phase. Additionally, we measure the benefit of inclusion text descriptions in the feature extractor training schema. Conclusion Bridge-Prompt and similar pre-trained+prompt-tuned video encoder models present significant visual representation for surgical robotics, especially in gesture recognition tasks. Given the diverse range of surgical tasks (gestures), the ability of these models to zero-shot transfer without the need for any task (gesture) specific retraining makes them invaluable.

Related papers

• An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos [32.257816070522885]
  We explore the possibility of using synthetically generated data for video-based gesture recognition with large pre-trained models. We use various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification. We find that using synthetic training videos yields significantly lower classification accuracy on real test videos compared to using a relatively small number of real training videos.
  arXiv  Detail & Related papers  (2024-10-03T02:31:14Z)
• Weakly-Supervised Surgical Phase Recognition [19.27227976291303]
  In this work we join concepts of graph segmentation with self-supervised learning to derive a random-walk solution for per-frame phase prediction. We validate our method by running experiments with the public Cholec80 dataset of laparoscopic cholecystectomy videos.
  arXiv  Detail & Related papers  (2023-10-26T07:54:47Z)
• Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures [51.78027546947034]
  Recent advancements in surgical computer vision have been driven by vision-only models, which lack language semantics. We propose leveraging surgical video lectures from e-learning platforms to provide effective vision and language supervisory signals. We address surgery-specific linguistic challenges using multiple automatic speech recognition systems for text transcriptions.
  arXiv  Detail & Related papers  (2023-07-27T22:38:12Z)
• A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision [93.90545426665999]
  We take a close look at autoregressive decoders for multi-task learning in multimodal computer vision. A key finding is that a small decoder learned on top of a frozen pretrained encoder works surprisingly well. It can be seen as teaching a decoder to interact with a pretrained vision model via natural language.
  arXiv  Detail & Related papers  (2023-03-30T13:42:58Z)
• DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training [73.74291217502928]
  We propose a simple framework, named DeCap, for zero-shot captioning. We introduce a lightweight visual-aware language decoder. We project the visual embedding into the CLIP text embedding space, while the projected embedding retains the information of the visual input.
  arXiv  Detail & Related papers  (2023-03-06T11:02:47Z)
• Pseudo-label Guided Cross-video Pixel Contrast for Robotic Surgical Scene Segmentation with Limited Annotations [72.15956198507281]
  We propose PGV-CL, a novel pseudo-label guided cross-video contrast learning method to boost scene segmentation. We extensively evaluate our method on a public robotic surgery dataset EndoVis18 and a public cataract dataset CaDIS.
  arXiv  Detail & Related papers  (2022-07-20T05:42:19Z)
• Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos [92.18898962396042]
  We propose a prompt-based framework, Bridge-Prompt, to model the semantics across adjacent actions. We reformulate the individual action labels as integrated text prompts for supervision, which bridge the gap between individual action semantics. Br-Prompt achieves state-of-the-art on multiple benchmarks.
  arXiv  Detail & Related papers  (2022-03-26T15:52:27Z)
• Prompting Visual-Language Models for Efficient Video Understanding [28.754997650215486]
  This paper presents a simple method to efficiently adapt one pre-trained visual-language model to novel tasks with minimal training. To bridge the gap between static images and videos, temporal information is encoded with lightweight Transformers stacking on top of frame-wise visual features.
  arXiv  Detail & Related papers  (2021-12-08T18:58:16Z)
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery [84.73764603474413]
  We propose a novel online approach of multi-modal graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset.
  arXiv  Detail & Related papers  (2020-11-03T11:00:10Z)
• Recurrent and Spiking Modeling of Sparse Surgical Kinematics [0.8458020117487898]
  A growing number of studies have used machine learning to analyze video and kinematic data captured from surgical robots. In this study, we explore the possibility of using only kinematic data to predict surgeons of similar skill levels. We report that it is possible to identify surgical fellows receiving near perfect scores in the simulation exercises based on their motion characteristics alone.
  arXiv  Detail & Related papers  (2020-05-12T15:41:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.