Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data

• URL: http://arxiv.org/abs/2501.15326v2
• Date: Tue, 06 May 2025 03:57:31 GMT
• Title: Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data
• Authors: Jiajie Li, Brian R Quaranto, Chenhui Xu, Ishan Mishra, Ruiyang Qin, Dancheng Liu, Peter C W Kim, Jinjun Xiong,
• Abstract summary: RASO is a foundation model designed to Recognize Any Surgical Object.<n>It generates tag-image-text pairs automatically from large-scale unannotated surgical lecture videos.<n>Our scalable data generation pipeline gathers 2,200 surgical procedures and produces 3.6 million tag annotations.
• Score: 15.00025814170182
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present RASO, a foundation model designed to Recognize Any Surgical Object, offering robust open-set recognition capabilities across a broad range of surgical procedures and object classes, in both surgical images and videos. RASO leverages a novel weakly-supervised learning framework that generates tag-image-text pairs automatically from large-scale unannotated surgical lecture videos, significantly reducing the need for manual annotations. Our scalable data generation pipeline gathers 2,200 surgical procedures and produces 3.6 million tag annotations across 2,066 unique surgical tags. Our experiments show that RASO achieves improvements of 2.9 mAP, 4.5 mAP, 10.6 mAP, and 7.2 mAP on four standard surgical benchmarks, respectively, in zero-shot settings, and surpasses state-of-the-art models in supervised surgical action recognition tasks. Code, model, and demo are available at https://ntlm1686.github.io/raso.

Related papers

• SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence [72.10889173696928]
  We propose SurgVLM, one of the first large vision-language foundation models for surgical intelligence.<n>We construct a large-scale multimodal surgical database, SurgVLM-DB, spanning more than 16 surgical types and 18 anatomical structures.<n>Building upon this comprehensive dataset, we propose SurgVLM, which is built upon Qwen2.5-VL, and undergoes instruction tuning to 10+ surgical tasks.
  arXiv  Detail & Related papers  (2025-06-03T07:44:41Z)
• Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance [50.486523249499115]
  Real-time video understanding is critical to guide procedures in minimally invasive surgery (MIS)<n>We propose Compress-to-Explore (C2E), a novel self-supervised framework to learn compact, informative representations from surgical videos.<n>C2E uses entropy-maximizing decoders to compress images while preserving clinically relevant details, improving encoder performance without labeled data.
  arXiv  Detail & Related papers  (2025-05-16T14:02:24Z)
• Surg-3M: A Dataset and Foundation Model for Perception in Surgical Settings [4.912213082028129]
  We present SurgFM, a self-supervised foundation model pretrained on Surg-3M that achieves impressive results in downstream tasks. Both Surg-3M and SurgFM have significant potential to accelerate progress towards developing autonomous robotic surgery systems.
  arXiv  Detail & Related papers  (2025-03-25T15:05:00Z)
• Identifying Surgical Instruments in Pedagogical Cataract Surgery Videos through an Optimized Aggregation Network [1.053373860696675]
  This paper presents a deep learning model for real-time identification of surgical instruments in cataract surgery videos.<n>Inspired by the architecture of YOLOV9, the model employs a Programmable Gradient Information (PGI) mechanism and a novel Generally-d Efficient Layer Aggregation Network (Go-ELAN)<n>The Go-ELAN YOLOV9 model, evaluated against YOLO v5, v7, v8, v9 vanilla, Laptool and DETR, achieves a superior mAP of 73.74 at IoU 0.5 on a dataset of 615 images.
  arXiv  Detail & Related papers  (2025-01-05T18:18:52Z)
• Is Segment Anything Model 2 All You Need for Surgery Video Segmentation? A Systematic Evaluation [25.459372606957736]
  In this paper, we systematically evaluate the performance of SAM2 model in zero-shot surgery video segmentation task.<n>We conducted experiments under different configurations, including different prompting strategies, robustness, etc.
  arXiv  Detail & Related papers  (2024-12-31T16:20:05Z)
• Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos [18.106255939686545]
  This letter presents a novel and real-time end-to-end error detection framework, Chain-of-Thought (COG) prompting. This encompasses two reasoning modules designed to mimic the decision-making processes of expert surgeons. Our method encapsulates the reasoning processes inherent to surgical activities enabling it to outperform the state-of-the-art by 4.6% in F1 score, 4.6% in Accuracy, and 5.9% in Jaccard index while processing each frame in 6.69 milliseconds on average.
  arXiv  Detail & Related papers  (2024-06-27T14:43:50Z)
• OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding [26.962250661485967]
  OphNet is a large-scale, expert-annotated video benchmark for ophthalmic surgical workflow understanding. A diverse collection of 2,278 surgical videos spanning 66 types of cataract, glaucoma, and corneal surgeries, with detailed annotations for 102 unique surgical phases and 150 fine-grained operations. OphNet is about 20 times larger than the largest existing surgical workflow analysis benchmark.
  arXiv  Detail & Related papers  (2024-06-11T17:18:11Z)
• OSSAR: Towards Open-Set Surgical Activity Recognition in Robot-assisted Surgery [13.843251369739908]
  We introduce an innovative Open-Set Surgical Activity Recognition (OSSAR) framework. Our solution leverages the hyperspherical reciprocal point strategy to enhance the distinction between known and unknown classes in the feature space. To support our assertions, we establish an open-set surgical activity benchmark utilizing the public JIGSAWS dataset.
  arXiv  Detail & Related papers  (2024-02-10T16:23:12Z)
• SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge [72.97934765570069]
  We release the first multimodal, publicly available, in-vivo, dataset for surgical action recognition and semantic instrumentation segmentation, containing 50 suturing video segments of Robotic Assisted Radical Prostatectomy (RARP) The aim of the challenge is to enable researchers to leverage the scale of the provided dataset and develop robust and highly accurate single-task action recognition and tool segmentation approaches in the surgical domain. A total of 12 teams participated in the challenge, contributing 7 action recognition methods, 9 instrument segmentation techniques, and 4 multitask approaches that integrated both action recognition and instrument segmentation.
  arXiv  Detail & Related papers  (2023-12-31T13:32:18Z)
• SAMSNeRF: Segment Anything Model (SAM) Guides Dynamic Surgical Scene Reconstruction by Neural Radiance Field (NeRF) [4.740415113160021]
  We propose a novel approach called SAMSNeRF that combines Segment Anything Model (SAM) and Neural Radiance Field (NeRF) techniques. Our experimental results on public endoscopy surgical videos demonstrate that our approach successfully reconstructs high-fidelity dynamic surgical scenes.
  arXiv  Detail & Related papers  (2023-08-22T20:31:00Z)
• Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures [51.78027546947034]
  Recent advancements in surgical computer vision have been driven by vision-only models, which lack language semantics. We propose leveraging surgical video lectures from e-learning platforms to provide effective vision and language supervisory signals. We address surgery-specific linguistic challenges using multiple automatic speech recognition systems for text transcriptions.
  arXiv  Detail & Related papers  (2023-07-27T22:38:12Z)
• GLSFormer : Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos [57.93194315839009]
  We propose a vision transformer-based approach to learn temporal features directly from sequence-level patches. We extensively evaluate our approach on two cataract surgery video datasets, Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-07-20T17:57:04Z)
• Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments [66.74633676595889]
  We present a multi-camera capture setup consisting of static and head-mounted cameras. Second, we publish a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured in a surgical wet lab and a real operating theatre. Third, we evaluate three state-of-the-art single-view and multi-view methods for the task of 6DoF pose estimation of surgical instruments.
  arXiv  Detail & Related papers  (2023-05-05T13:42:19Z)
• Dissecting Self-Supervised Learning Methods for Surgical Computer Vision [51.370873913181605]
  Self-Supervised Learning (SSL) methods have begun to gain traction in the general computer vision community. The effectiveness of SSL methods in more complex and impactful domains, such as medicine and surgery, remains limited and unexplored. We present an extensive analysis of the performance of these methods on the Cholec80 dataset for two fundamental and popular tasks in surgical context understanding, phase recognition and tool presence detection.
  arXiv  Detail & Related papers  (2022-07-01T14:17:11Z)
• CholecTriplet2021: A benchmark challenge for surgical action triplet recognition [66.51610049869393]
  This paper presents CholecTriplet 2021: an endoscopic vision challenge organized at MICCAI 2021 for the recognition of surgical action triplets in laparoscopic videos. We present the challenge setup and assessment of the state-of-the-art deep learning methods proposed by the participants during the challenge. A total of 4 baseline methods and 19 new deep learning algorithms are presented to recognize surgical action triplets directly from surgical videos, achieving mean average precision (mAP) ranging from 4.2% to 38.1%.
  arXiv  Detail & Related papers  (2022-04-10T18:51:55Z)
• Towards Unified Surgical Skill Assessment [18.601526803020885]
  We propose a unified multi-path framework for automatic surgical skill assessment. We conduct experiments on the JIGSAWS dataset of simulated surgical tasks, and a new clinical dataset of real laparoscopic surgeries.
  arXiv  Detail & Related papers  (2021-06-02T09:06:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.