CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection

• URL: http://arxiv.org/abs/2410.07540v1
• Date: Thu, 10 Oct 2024 02:22:09 GMT
• Title: CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection
• Abstract summary: We introduce a multi-level surgical motion dataset (CoPESD) for training Large Visual-Language Models (LVLMs) as the robotic textbfCo-textbfPilot of textbfEndoscopic textbfSubmucosal textbfDissection. CoPESD includes 17,679 images with 32,699 bounding boxes and 88,395 multi-level motions, from over 35 hours of ESD videos for both robot-assisted and conventional surgeries.
• Score: 41.51336459978475
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: submucosal dissection (ESD) enables rapid resection of large lesions, minimizing recurrence rates and improving long-term overall survival. Despite these advantages, ESD is technically challenging and carries high risks of complications, necessitating skilled surgeons and precise instruments. Recent advancements in Large Visual-Language Models (LVLMs) offer promising decision support and predictive planning capabilities for robotic systems, which can augment the accuracy of ESD and reduce procedural risks. However, existing datasets for multi-level fine-grained ESD surgical motion understanding are scarce and lack detailed annotations. In this paper, we design a hierarchical decomposition of ESD motion granularity and introduce a multi-level surgical motion dataset (CoPESD) for training LVLMs as the robotic \textbf{Co}-\textbf{P}ilot of \textbf{E}ndoscopic \textbf{S}ubmucosal \textbf{D}issection. CoPESD includes 17,679 images with 32,699 bounding boxes and 88,395 multi-level motions, from over 35 hours of ESD videos for both robot-assisted and conventional surgeries. CoPESD enables granular analysis of ESD motions, focusing on the complex task of submucosal dissection. Extensive experiments on the LVLMs demonstrate the effectiveness of CoPESD in training LVLMs to predict following surgical robotic motions. As the first multimodal ESD motion dataset, CoPESD supports advanced research in ESD instruction-following and surgical automation. The dataset is available at \href{https://github.com/gkw0010/CoPESD}{https://github.com/gkw0010/CoPESD.}}

Related papers

• Essential Subspace Merging for Multi-Task Learning [78.5390284258307]
  We analyze the output shifts induced by task updates and observe that their energy is concentrated in a small number of principal directions.<n>Most remaining directions carry little task-relevant energy, but their accumulation across multiple task updates can cause severe interference during merging.<n>Based on ESD, we introduce Essential Subspace Merging (ESM), a training-free static merging method.<n>We extend ESM to ESM++, a training-free dynamic merging method that decomposes task-specific residuals into low-rank experts.
  arXiv  Detail & Related papers  (2026-06-17T15:07:11Z)
• SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding [75.00667948967848]
  The SurgLLM framework is a large multimodal model tailored for versatile surgical video understanding tasks.<n>To empower the spatial focus of surgical videos, we first devise Surgical Context-aware Multimodal Pretraining (Surg-Pretrain) for the video encoder of SurgLLM.<n>To incorporate surgical temporal knowledge into SurgLLM, we further propose Temporal-aware Multimodal Tuning (TM-Tuning) to enhance temporal reasoning with interleaved multimodal embeddings.
  arXiv  Detail & Related papers  (2025-08-30T04:36:41Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• Few-Shot Deployment of Pretrained MRI Transformers in Brain Imaging Tasks [2.982793366290863]
  We propose a framework for the few-shot deployment of pretrained MRI transformers in diverse brain imaging tasks.<n>By utilizing the Masked Autoencoder (MAE) pretraining strategy, we obtain highly transferable latent representations that generalize well across tasks and datasets.
  arXiv  Detail & Related papers  (2025-08-07T18:53:28Z)
• HMSViT: A Hierarchical Masked Self-Supervised Vision Transformer for Corneal Nerve Segmentation and Diabetic Neuropathy Diagnosis [3.8141400767898603]
  Diabetic Peripheral Neuropathy (DPN) affects nearly half of diabetes patients, requiring early detection.<n>We propose HMSViT, a novel Hierarchical Masked Self-Supervised Vision Transformer (HMSViT)<n>HMSViT employs pooling-based hierarchical and dual attention mechanisms with absolute positional encoding, enabling efficient multi-scale feature extraction.<n> Experiments on clinical CCM datasets showed HMSViT achieves state-of-the-art performance, with 61.34% mIoU for nerve segmentation and 70.40% diagnostic accuracy.
  arXiv  Detail & Related papers  (2025-06-24T10:00:23Z)
• SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model [55.13206879750197]
  SurgVidLM is the first video language model designed to address both full and fine-grained surgical video comprehension.<n>We introduce the StageFocus mechanism which is a two-stage framework performing the multi-grained, progressive understanding of surgical videos.<n> Experimental results demonstrate that SurgVidLM significantly outperforms state-of-the-art Vid-LLMs in both full and fine-grained video understanding tasks.
  arXiv  Detail & Related papers  (2025-06-22T02:16:18Z)
• $\mathsf{CSMAE~}$:~Cataract Surgical Masked Autoencoder (MAE) based Pre-training [25.71088804562768]
  We introduce a Masked Autoencoder (MAE)-based pretraining approach, specifically developed for Cataract Surgery video analysis. Instead of randomly selecting tokens for masking they are selected based on the importance of the token token. This approach surpasses current state-of-the-art self-supervised pretraining and adapter-based learning methods by a significant margin.
  arXiv  Detail & Related papers  (2025-02-12T22:24:49Z)
• AMNCutter: Affinity-Attention-Guided Multi-View Normalized Cutter for Unsupervised Surgical Instrument Segmentation [7.594796294925481]
  We propose a label-free unsupervised model featuring a novel module named Multi-View Normalized Cutter (m-NCutter) Our model is trained using a graph-cutting loss function that leverages patch affinities for supervision, eliminating the need for pseudo-labels. We conduct comprehensive experiments across multiple SIS datasets to validate our approach's state-of-the-art (SOTA) performance, robustness, and exceptional potential as a pre-trained model.
  arXiv  Detail & Related papers  (2024-11-06T06:33:55Z)
• Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation [51.222684687924215]
  Surgical video-language pretraining faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data. We propose a hierarchical knowledge augmentation approach and a novel Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining framework to tackle these issues.
  arXiv  Detail & Related papers  (2024-09-30T22:21:05Z)
• PitRSDNet: Predicting Intra-operative Remaining Surgery Duration in Endoscopic Pituitary Surgery [7.291847156946912]
  This paper presents PitRSDNet for predicting Remaining Surgery Duration (RSD) during pituitary surgery. PitRSDNet integrates workflow knowledge into RSD prediction in two forms: 1) multi-task learning for concurrently predicting step and RSD; and 2) prior steps as context in temporal learning and inference. PitRSDNet is trained and evaluated on a new endoscopic pituitary surgery dataset with 88 videos to show competitive performance improvements over previous statistical and machine learning methods.
  arXiv  Detail & Related papers  (2024-09-25T15:03:22Z)
• SPRMamba: Surgical Phase Recognition for Endoscopic Submucosal Dissection with Mamba [6.531066045206769]
  We present SPRMamba, a novel framework for real-time surgical phase recognition.<n>It integrates a Mamba architecture with a Scaled Residual TranMamba block to synergize temporal modeling and localized detail extraction.<n>It achieves state-of-the-art performance (87.64% accuracy on ESD385, +1.0% over prior methods) demonstrating robust generalizability across surgical procedures.
  arXiv  Detail & Related papers  (2024-09-18T16:26:56Z)
• Enhancing Weakly Supervised 3D Medical Image Segmentation through Probabilistic-aware Learning [52.249748801637196]
  3D medical image segmentation is a challenging task with crucial implications for disease diagnosis and treatment planning. Recent advances in deep learning have significantly enhanced fully supervised medical image segmentation. We propose a novel probabilistic-aware weakly supervised learning pipeline, specifically designed for 3D medical imaging.
  arXiv  Detail & Related papers  (2024-03-05T00:46:53Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• Diagnosing Alzheimer's Disease using Early-Late Multimodal Data Fusion with Jacobian Maps [1.5501208213584152]
  Alzheimer's disease (AD) is a prevalent and debilitating neurodegenerative disorder impacting a large aging population. We propose an efficient early-late fusion (ELF) approach, which leverages a convolutional neural network for automated feature extraction and random forests. To tackle the challenge of detecting subtle changes in brain volume, we transform images into the Jacobian domain (JD)
  arXiv  Detail & Related papers  (2023-10-25T19:02:57Z)
• Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues [52.886545681833596]
  LerPlane is a novel method for fast and accurate reconstruction of surgical scenes under a single-viewpoint setting. LerPlane treats surgical procedures as 4D volumes and factorizes them into explicit 2D planes of static and dynamic fields. LerPlane shares static fields, significantly reducing the workload of dynamic tissue modeling.
  arXiv  Detail & Related papers  (2023-05-31T14:38:35Z)
• Robotic Navigation Autonomy for Subretinal Injection via Intelligent Real-Time Virtual iOCT Volume Slicing [88.99939660183881]
  We propose a framework for autonomous robotic navigation for subretinal injection. Our method consists of an instrument pose estimation method, an online registration between the robotic and the i OCT system, and trajectory planning tailored for navigation to an injection target. Our experiments on ex-vivo porcine eyes demonstrate the precision and repeatability of the method.
  arXiv  Detail & Related papers  (2023-01-17T21:41:21Z)
• Brain Lesion Synthesis via Progressive Adversarial Variational Auto-Encoder [0.9954435559869312]
  Region of interest (ROI) segmentation before and after laser interstitial thermal therapy (LITT) would enable automated lesion quantification. CNNs are state-of-the-art solutions for ROI segmentation, but require large amounts of annotated data during the training. We propose a progressive brain lesion synthesis framework (PAVAE) to expand both the quantity and diversity of the training dataset.
  arXiv  Detail & Related papers  (2022-08-05T14:39:06Z)
• External Attention Assisted Multi-Phase Splenic Vascular Injury Segmentation with Limited Data [72.99534552950138]
  The spleen is one of the most commonly injured solid organs in blunt abdominal trauma. accurate segmentation of splenic vascular injury is challenging for the following reasons.
  arXiv  Detail & Related papers  (2022-01-04T02:35:56Z)
• The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods [15.833413083110903]
  This paper presents ESAD, the first large-scale dataset designed to tackle the problem of surgeon action detection in endoscopic minimally invasive surgery. The dataset provides bounding box annotation for 21 action classes on real endoscopic video frames captured during prostatectomy, and was used as the basis of a recent MIDL 2020 challenge.
  arXiv  Detail & Related papers  (2021-04-07T15:11:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.