Scaling Video Pretraining for Surgical Foundation Models

• URL: http://arxiv.org/abs/2603.29966v2
• Date: Thu, 02 Apr 2026 16:46:06 GMT
• Title: Scaling Video Pretraining for Surgical Foundation Models
• Abstract summary: SurgRec is a scalable and reproducible pretraining recipe for surgical video understanding.<n>We curate a large corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery.
• Score: 51.92777479821822
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Surgical video understanding is essential for computer-assisted interventions, yet existing surgical foundation models remain constrained by limited data scale, procedural diversity, and inconsistent evaluation, often lacking a reproducible training pipeline. We propose SurgRec, a scalable and reproducible pretraining recipe for surgical video understanding, instantiated with two variants: SurgRec-MAE and SurgRec-JEPA. We curate a large multi-source corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery. Building on this corpus, we develop a unified pretraining pipeline with balanced sampling and standardize a reproducible benchmark across 16 downstream datasets and four clinical domains with consistent data splits. Across extensive comparisons against SSL baselines and vision-language models, SurgRec consistently achieves superior performance across downstream datasets. In contrast, VLMs prove unreliable for fine-grained temporal recognition, exhibiting both performance gaps and sensitivity to prompt phrasing. Our work provides a reproducible, scalable foundation for the community to build more general surgical video models. All code, models, and data will be publicly released.

Related papers

• Hi-GaTA: Hierarchical Gated Temporal Aggregation Adapter for Surgical Video Report Generation [7.606404030331724]
  We propose a Perception-Reasoning framework for surgical report generation, featuring HiGa-TA, a lightweight temporal adapter.<n>Experiments show our approach achieves the best overall performance, with consistent gains over strong Multigrained Large Language Model (MLLM) baselines.
  arXiv  Detail & Related papers  (2026-05-11T20:21:34Z)
• UniSurgSAM: A Unified Promptable Model for Reliable Surgical Video Segmentation [18.74680721916099]
  We present UniSurgSAM, a unified PVOS model enabling reliable surgical video segmentation through visual, textual, or audio prompts.<n>We introduce three key designs for reliability: presence-aware decoding that targets absence to suppress hallucinations; boundary-aware long-term tracking that prevents mask drift over extended sequences; and adaptive state transition that closes the loop between stages for failure recovery.<n>UniSurgSAM achieves state-of-the-art performance in real time across all prompt modalities and granularities, providing a practical foundation for computer-assisted surgery.
  arXiv  Detail & Related papers  (2026-04-04T08:44:10Z)
• A benchmark for video-based laparoscopic skill analysis and assessment [1.5734501497837607]
  We introduce the Laparoscopic Skill Analysis and Assessment dataset, comprising 1270 stereo video recordings of four basic laparoscopic training tasks.<n>Each recording is annotated with a structured skill rating, aggregated from three independent raters, as well as binary labels indicating the presence or absence of task-specific errors.<n>To facilitate benchmarking of both existing and novel approaches for video-based skill assessment and error recognition, we provide predefined data splits for each task.
  arXiv  Detail & Related papers  (2026-02-10T15:59:19Z)
• UniSurg: A Video-Native Foundation Model for Universal Understanding of Surgical Videos [81.9180187964947]
  We present UniSurg, a foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction.<n>To enable large-scale pretraining, we curate the largest surgical video dataset to date, comprising 3,658 hours of video from 50 sources across 13 anatomical regions.<n>These results establish UniSurg as a new standard for universal, motion-oriented surgical video understanding.
  arXiv  Detail & Related papers  (2026-02-05T13:18:33Z)
• Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model [7.746379804154433]
  $SurgiFlowVid$ is a sparse and controllable video diffusion framework for generating surgical videos of under-represented classes.<n>We validate our approach on three surgical datasets across tasks including action recognition, tool presence detection, and laparoscope motion prediction.
  arXiv  Detail & Related papers  (2025-10-07T20:29:27Z)
• Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis [0.7068165275780814]
  We use V-JEPA as the single-modality foundation of a multimodal model for minimally invasive surgery support.<n>We analyze how the model's downstream performance can benefit from finetuning on unlabeled surgical video data.<n>Our results thus demonstrate how surgical data science can leverage public, generic foundation models.
  arXiv  Detail & Related papers  (2025-09-08T16:04:19Z)
• SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model [67.8359850515282]
  SurgVidLM is the first video language model designed to address both full and fine-grained surgical video comprehension.<n>We show that SurgVidLM significantly outperforms state-of-the-art Vid-LLMs of comparable parameter scale in both full and fine-grained video understanding tasks.
  arXiv  Detail & Related papers  (2025-06-22T02:16:18Z)
• Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance [50.486523249499115]
  Real-time video understanding is critical to guide procedures in minimally invasive surgery (MIS)<n>We propose Compress-to-Explore (C2E), a novel self-supervised framework to learn compact, informative representations from surgical videos.<n>C2E uses entropy-maximizing decoders to compress images while preserving clinically relevant details, improving encoder performance without labeled data.
  arXiv  Detail & Related papers  (2025-05-16T14:02:24Z)
• Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models [1.5678321653327674]
  We propose a two-stage, text-based method to generate high-fidelity surgical videos for under-represented classes.<n>We evaluate our method on two downstream tasks--action recognition and intra-operative event prediction-demonstrating.
  arXiv  Detail & Related papers  (2025-05-14T23:43:29Z)
• SASVi - Segment Any Surgical Video [2.330834737588252]
  We propose SASVi, a novel re-prompting mechanism based on a frame-wise Mask R-CNN Overseer model.<n>This model automatically re-prompts the foundation model SAM2 when the scene constellation changes.
  arXiv  Detail & Related papers  (2025-02-12T00:29:41Z)
• Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime [70.04389979779195]
  This paper explores training medical vision-language models (VLMs) where the visual and language inputs are embedded into a common space. We explore several candidate methods to improve low-data performance, including adapting generic pre-trained models to novel image and text domains. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports.
  arXiv  Detail & Related papers  (2023-03-30T18:20:00Z)
• LRTD: Long-Range Temporal Dependency based Active Learning for Surgical Workflow Recognition [67.86810761677403]
  We propose a novel active learning method for cost-effective surgical video analysis. Specifically, we propose a non-local recurrent convolutional network (NL-RCNet), which introduces non-local block to capture the long-range temporal dependency. We validate our approach on a large surgical video dataset (Cholec80) by performing surgical workflow recognition task.
  arXiv  Detail & Related papers  (2020-04-21T09:21:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.