SHANDS: A Multi-View Dataset and Benchmark for Surgical Hand-Gesture and Error Recognition Toward Medical Training

• URL: http://arxiv.org/abs/2603.26400v1
• Date: Fri, 27 Mar 2026 13:24:41 GMT
• Title: SHANDS: A Multi-View Dataset and Benchmark for Surgical Hand-Gesture and Error Recognition Toward Medical Training
• Abstract summary: Surgical-Hands is a large-scale multi-view video dataset for surgical hand-gesture and error recognition for medical training.<n>Video is annotated at the frame level with 15 gesture primitives and includes a validated taxonomy of 8 trainee error types.<n>SHands is publicly released to support the development of robust and scalable AI systems for surgical training.
• Score: 1.9550493958421307
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In surgical training for medical students, proficiency development relies on expert-led skill assessment, which is costly, time-limited, difficult to scale, and its expertise remains confined to institutions with available specialists. Automated AI-based assessment offers a viable alternative, but progress is constrained by the lack of datasets containing realistic trainee errors and the multi-view variability needed to train robust computer vision approaches. To address this gap, we present Surgical-Hands (SHands), a large-scale multi-view video dataset for surgical hand-gesture and error recognition for medical training. \textsc{SHands} captures linear incision and suturing using five RGB cameras from complementary viewpoints, performed by 52 participants (20 experts and 32 trainees), each completing three standardized trials per procedure. The videos are annotated at the frame level with 15 gesture primitives and include a validated taxonomy of 8 trainee error types, enabling both gesture recognition and error detection. We further define standardized evaluation protocols for single-view, multi-view, and cross-view generalization, and benchmark state-of-the-art deep learning models on the dataset. SHands is publicly released to support the development of robust and scalable AI systems for surgical training grounded in clinically curated domain knowledge.

Related papers

• OSS: Open Suturing Skills Vision-Based Assessment Challenge 2024-2025 [15.333165075116964]
  We present the results of a dedicated MII challenge designed to benchmark and advance vision-based skill assessment in open surgery.<n>The challenge dataset comprises of an open suturing training task recorded with a static GoPro camera in a dry-lab setting.<n>Participants submitted diverse solutions including deep learning-based video models, tracking-driven methods, and hybrid approaches.
  arXiv  Detail & Related papers  (2026-05-21T09:04:17Z)
• A generalizable foundation model for intraoperative understanding across surgical procedures [1.0412442875956527]
  We introduce ZEN, a generalizable foundation model for intraoperative surgical video understanding trained on more than 4 million frames from over 21 procedures.<n>ZEN consistently outperforms existing surgical foundation models and demonstrates robust cross-procedure generalization.
  arXiv  Detail & Related papers  (2026-02-14T06:52:42Z)
• A benchmark for video-based laparoscopic skill analysis and assessment [1.5734501497837607]
  We introduce the Laparoscopic Skill Analysis and Assessment dataset, comprising 1270 stereo video recordings of four basic laparoscopic training tasks.<n>Each recording is annotated with a structured skill rating, aggregated from three independent raters, as well as binary labels indicating the presence or absence of task-specific errors.<n>To facilitate benchmarking of both existing and novel approaches for video-based skill assessment and error recognition, we provide predefined data splits for each task.
  arXiv  Detail & Related papers  (2026-02-10T15:59:19Z)
• DermINO: Hybrid Pretraining for a Versatile Dermatology Foundation Model [92.66916452260553]
  DermNIO is a versatile foundation model for dermatology.<n>It incorporates a novel hybrid pretraining framework that augments the self-supervised learning paradigm.<n>It consistently outperforms state-of-the-art models across a wide range of tasks.
  arXiv  Detail & Related papers  (2025-08-17T00:41:39Z)
• ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection [54.270188252068145]
  ProstaTD is a large-scale dataset for surgical triplet detection developed from the technically demanding domain of robot-assisted prostatectomy.<n>The dataset comprises 71,775 video frames and 196,490 annotated triplet instances, collected from 21 surgeries performed across multiple institutions.<n>ProstaTD is the largest and most diverse surgical triplet dataset to date, moving the field from simple classification to full detection with precise spatial and temporal boundaries.
  arXiv  Detail & Related papers  (2025-06-01T19:29:39Z)
• Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers [1.194275822303467]
  Vision Transformer for irregular sampled Multi-modal Measurements (ViTiMM)<n>Our approach, Vision Transformer for irregular sampled Multi-modal Measurements (ViTiMM), not only simplifies data preprocessing and modeling but also outperforms current state-of-the-art methods in predicting in-hospital mortality and phenotyping, as evaluated on 6,175 patients from the MIMIC-IV dataset.<n>We hope our work inspires advancements in multi-modal medical AI by reducing the training complexity to (visual) prompt engineering, thus lowering entry barriers and enabling no-code solutions for training.
  arXiv  Detail & Related papers  (2025-01-30T09:52:15Z)
• CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Tools [1.7059333957102913]
  Existing datasets rely on overly generic tracking formalizations that fail to capture surgical-specific dynamics.<n>We introduce CholecTrack20, a specialized dataset for multi-class, multi-tool tracking in surgical procedures.<n>The dataset comprises 20 full-length surgical videos, annotated at 1 fps, yielding over 35K frames and 65K labeled tool instances.
  arXiv  Detail & Related papers  (2023-12-12T15:18:15Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• Dissecting Self-Supervised Learning Methods for Surgical Computer Vision [51.370873913181605]
  Self-Supervised Learning (SSL) methods have begun to gain traction in the general computer vision community. The effectiveness of SSL methods in more complex and impactful domains, such as medicine and surgery, remains limited and unexplored. We present an extensive analysis of the performance of these methods on the Cholec80 dataset for two fundamental and popular tasks in surgical context understanding, phase recognition and tool presence detection.
  arXiv  Detail & Related papers  (2022-07-01T14:17:11Z)
• When Accuracy Meets Privacy: Two-Stage Federated Transfer Learning Framework in Classification of Medical Images on Limited Data: A COVID-19 Case Study [77.34726150561087]
  COVID-19 pandemic has spread rapidly and caused a shortage of global medical resources. CNN has been widely utilized and verified in analyzing medical images.
  arXiv  Detail & Related papers  (2022-03-24T02:09:41Z)
• Robust Medical Instrument Segmentation Challenge 2019 [56.148440125599905]
  Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. Our challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap.
  arXiv  Detail & Related papers  (2020-03-23T14:35:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.