SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery

• URL: http://arxiv.org/abs/2606.25905v1
• Date: Wed, 24 Jun 2026 14:53:56 GMT
• Title: SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery
• Abstract summary: We introduce SurgAtlas, the largest surgical video-language dataset to date, comprising 15,291 videos (2,391 hours) spanning 18 surgical specialties and over 5,000 procedure types.<n> SurgAtlas is also the first surgical video-language dataset to include open surgery at scale, with 6,182 open procedure videos alongside over 9,000 minimally invasive recordings.<n>We additionally provide an expert-validated subset with verified visual question-answer pairs across diverse open and minimally invasive procedures, serving as a clinically grounded benchmark for surgical reasoning.
• Score: 33.663697869976
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce SurgAtlas, the largest surgical video-language dataset to date, comprising 15,291 videos (2,391 hours) spanning 18 surgical specialties and over 5,000 procedure types, sourced entirely from publicly available YouTube content. SurgAtlas is also the first surgical video-language dataset to include open surgery at scale, with 6,182 open procedure videos alongside over 9,000 minimally invasive recordings, and the first to establish standardized benchmarks for open-surgery video understanding. We additionally provide an expert-validated subset with verified visual question-answer pairs across diverse open and minimally invasive procedures, serving as a clinically grounded benchmark for surgical reasoning. Compared with existing surgical video-language datasets, SurgAtlas provides one of the most diverse annotation schemas, combining segment-level captions, step- and phase-level descriptions, video-level surgical descriptions, and reasoning-oriented question-answer pairs organized within a hierarchical taxonomy. These annotations are constructed through an automated multi-tier pipeline with LLM-based enrichment and a staged VQA generation framework with explicit groundedness verification. The scale and diversity of SurgAtlas enable training surgical foundation models with broad procedural coverage: we finetune Qwen3-VL-8B through a two-stage captioning-then-instruction pipeline and achieve competitive or state-of-the-art results on multiple established surgical benchmarks, including phase recognition, triplet detection, and reasoning question answering. More broadly, SurgAtlas provides a large native public video corpus that can support future large-scale pretraining of multimodal surgical AI systems and contribute to the development of next-generation foundation models for surgery.

Related papers

• A vision-language model and platform for temporally mapping surgery from video [2.456019890058713]
  Mapping surgery is fundamental to developing operative guidelines and enabling autonomous robotic surgery.<n>Recent advances in artificial intelligence (AI) have shown promise in mapping the behaviour of surgeons from videos.<n>Here we introduce Halsted, a vision-language model trained on the Halsted Surgical Atlas.
  arXiv  Detail & Related papers  (2026-03-23T21:23:40Z)
• UniSurg: A Video-Native Foundation Model for Universal Understanding of Surgical Videos [81.9180187964947]
  We present UniSurg, a foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction.<n>To enable large-scale pretraining, we curate the largest surgical video dataset to date, comprising 3,658 hours of video from 50 sources across 13 anatomical regions.<n>These results establish UniSurg as a new standard for universal, motion-oriented surgical video understanding.
  arXiv  Detail & Related papers  (2026-02-05T13:18:33Z)
• SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding [75.00667948967848]
  The SurgLLM framework is a large multimodal model tailored for versatile surgical video understanding tasks.<n>To empower the spatial focus of surgical videos, we first devise Surgical Context-aware Multimodal Pretraining (Surg-Pretrain) for the video encoder of SurgLLM.<n>To incorporate surgical temporal knowledge into SurgLLM, we further propose Temporal-aware Multimodal Tuning (TM-Tuning) to enhance temporal reasoning with interleaved multimodal embeddings.
  arXiv  Detail & Related papers  (2025-08-30T04:36:41Z)
• HieraSurg: Hierarchy-Aware Diffusion Model for Surgical Video Generation [44.37374628674769]
  We propose HieraSurg, a hierarchy-aware surgical video generation framework consisting of two specialized diffusion models.<n>The model exhibits particularly fine-grained adherence when provided with existing segmentation maps, suggesting its potential for practical surgical applications.
  arXiv  Detail & Related papers  (2025-06-26T14:07:23Z)
• SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model [67.8359850515282]
  SurgVidLM is the first video language model designed to address both full and fine-grained surgical video comprehension.<n>We show that SurgVidLM significantly outperforms state-of-the-art Vid-LLMs of comparable parameter scale in both full and fine-grained video understanding tasks.
  arXiv  Detail & Related papers  (2025-06-22T02:16:18Z)
• SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysis [20.566701996432226]
  SurgBench is a unified surgical video benchmarking framework comprising a pretraining dataset, textbfSurgBench-P, and an evaluation benchmark, textbfSurgBench-E.<n>SurgBench-P covers 53 million frames across 22 surgical procedures and 11 specialties, and SurgBench-E provides robust evaluation across six categories (phase classification, camera motion, tool recognition, disease diagnosis, action classification, and organ detection) spanning 72 fine-grained tasks.
  arXiv  Detail & Related papers  (2025-06-09T10:02:58Z)
• SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence [72.10889173696928]
  We propose SurgVLM, one of the first large vision-language foundation models for surgical intelligence.<n>We construct a large-scale multimodal surgical database, SurgVLM-DB, spanning more than 16 surgical types and 18 anatomical structures.<n>Building upon this comprehensive dataset, we propose SurgVLM, which is built upon Qwen2.5-VL, and undergoes instruction tuning to 10+ surgical tasks.
  arXiv  Detail & Related papers  (2025-06-03T07:44:41Z)
• OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining [60.75854609803651]
  OphCLIP is a hierarchical retrieval-augmented vision-language pretraining framework for ophthalmic surgical workflow understanding.<n>OphCLIP learns both fine-grained and long-term visual representations by aligning short video clips with detailed narrative descriptions and full videos with structured titles.<n>Our OphCLIP also designs a retrieval-augmented pretraining framework to leverage the underexplored large-scale silent surgical procedure videos.
  arXiv  Detail & Related papers  (2024-11-23T02:53:08Z)
• LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning [15.646322352232819]
  We create a new dataset, Surg-QA, consisting of 102,000 surgical video-instruction pairs. We propose a novel two-stage question-answer generation pipeline with LLM to learn surgical knowledge. We train LLaVA-Surg, a novel vision-language conversational assistant capable of answering open-ended questions about surgical videos.
  arXiv  Detail & Related papers  (2024-08-15T07:00:20Z)
• Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures [50.09187683845788]
  Recent advancements in surgical computer vision applications have been driven by vision-only models.<n>These methods rely on manually annotated surgical videos to predict a fixed set of object categories.<n>In this work, we put forward the idea that the surgical video lectures available through open surgical e-learning platforms can provide effective vision and language supervisory signals.
  arXiv  Detail & Related papers  (2023-07-27T22:38:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.