SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos

• URL: http://arxiv.org/abs/2604.09037v1
• Date: Fri, 10 Apr 2026 06:58:29 GMT
• Title: SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos
• Abstract summary: We introduce SiMing-Bench, the first benchmark for evaluating this capability from full-length clinical skill videos.<n>It targets rubric-grounded process-level judgment of whether interaction-driven state updates preserve procedural correctness across an entire workflow.
• Score: 33.33285554449996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current video benchmarks for multimodal large language models (MLLMs) focus on event recognition, temporal ordering, and long-context recall, but overlook a harder capability required for expert procedural judgment: tracking how ongoing interactions update the procedural state and thereby determine the correctness of later actions. We introduce SiMing-Bench, the first benchmark for evaluating this capability from full-length clinical skill videos. It targets rubric-grounded process-level judgment of whether interaction-driven state updates preserve procedural correctness across an entire workflow. SiMing-Bench is instantiated with SiMing-Score, a physician-annotated dataset of real clinical skill examination videos spanning cardiopulmonary resuscitation, automated external defibrillator operation, and bag-mask ventilation, each paired with a standardized step-wise rubric and dual-expert labels. Across diverse open- and closed-source MLLMs, we observe consistently weak agreement with physician judgments. Moreover, weak performance on rubric-defined intermediate steps persists even when overall procedure-level correlation appears acceptable, suggesting that coarse global assessment substantially overestimates current models' procedural judgment ability. Additional analyses with binary step judgment and step-aligned clips indicate that the bottleneck is not merely fine-grained scoring or temporal localization, but modeling how continuous interactions update procedural state over time.

Related papers

• StrokeGuard: A Multi-Agent Guided System for Prehospital Stroke Assessment [3.046219182166974]
  In clinical practice, FAST-based scales are widely used for prehospital stroke assessment.<n>Non-clinical users often encounter challenges such as inaccurate descriptions, incomplete symptom observation, and difficult operational procedures.<n>This paper presents StrokeGuard: a multi-agent guided system designed for prehospital stroke assessment.
  arXiv  Detail & Related papers  (2026-08-25T13:39:30Z)
• ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents [14.434996950038956]
  ClinEnv scores both what the model decides, through deterministic-grounded matching, and how it gathers information.<n> ClinEnv makes this information-acquisition gap, invisible to outcome-only evaluation, directly measurable.
  arXiv  Detail & Related papers  (2026-06-01T17:56:26Z)
• IMPACT-Scribe: Interactive Temporal Action Segmentation with Boundary Scribbles and Query Planning [75.72434806468884]
  IMPACT-Scribe is a correction-driven framework for dense labeling.<n>It combines uncertainty-aware boundary supervision, local proposal modeling, cost-aware query planning, structured propagation, and correction-driven adaptation.<n> Experiments and a human study show that this closed-loop design improves labeling quality per effort, enhances boundary accuracy, and fosters better humanmachine interaction over time.
  arXiv  Detail & Related papers  (2026-05-03T01:45:58Z)
• SurgPhase: Time efficient pituitary tumor surgery phase recognition via an interactive web platform [4.1977714530157435]
  We present a comprehensive framework for phase recognition in pituitary tumor surgery (PTS) videos.<n>Our method achieves 90% accuracy on a held-out test set, outperforming current state-of-the-art approaches.<n>A central contribution of this work is the integration of a collaborative online platform designed for surgeons to upload surgical videos.
  arXiv  Detail & Related papers  (2026-03-26T00:22:20Z)
• ClinConsensus: A Consensus-Based Benchmark for Evaluating Chinese Medical LLMs across Difficulty Levels [39.33170904610862]
  Large language models (LLMs) are increasingly applied to health management, showing promise across disease prevention, clinical decision-making, and long-term care.<n>We introduce ClinConsensus, a Chinese medical benchmark curated, validated and quality-controlled by clinical experts.<n> ClinConsensus comprises 2500 open-ended cases spanning the full continuum of care--from prevention and intervention to long-term follow-up--covering 36 medical specialties, 12 common clinical task types, and progressively increasing levels of complexity.
  arXiv  Detail & Related papers  (2026-03-02T17:17:18Z)
• MMTA: Multi Membership Temporal Attention for Fine-Grained Stroke Rehabilitation Assessment [0.28332284137397873]
  We introduce Multi-Membership Temporal Attention (MMTA), a high-resolution temporal transformer for fine-grained rehabilitation assessment.<n>MMTA supports both video and wearable IMU inputs within a unified single-stage architecture.<n>It consistently improves over the Global Attention transformer, boosting Edit Score by +1.3 (Video) and +1.6 (IMU) on StrokeRehab while further improving 50Salads by +3.3.
  arXiv  Detail & Related papers  (2026-03-01T02:57:03Z)
• MedConsultBench: A Full-Cycle, Fine-Grained, Process-Aware Benchmark for Medical Consultation Agents [10.109613967215447]
  We propose MedConsultBench, a comprehensive framework designed to evaluate the complete online consultation cycle.<n>Our methodology introduces Atomic Information Units (AIUs) to track clinical information acquisition at a sub-turn level.<n>By addressing the underspecification and ambiguity inherent in online consultations, the benchmark evaluates uncertainty-aware yet concise inquiry.
  arXiv  Detail & Related papers  (2026-01-19T02:18:10Z)
• A Stitch in Time: Learning Procedural Workflow via Self-Supervised Plackett-Luce Ranking [11.039713164587456]
  Procedural activities are highly structured as a set of actions conducted in a specific temporal order.<n>Current self-supervised learning methods often overlook the procedural nature that underpins such activities.<n>We propose PL-Stitch, a self-supervised framework that harnesses the inherent temporal order of video frames as a powerful supervisory signal.
  arXiv  Detail & Related papers  (2025-11-21T21:59:22Z)
• GLSFormer : Gated - Long, Short Sequence Transformer for Step Recognition in Surgical Videos [57.93194315839009]
  We propose a vision transformer-based approach to learn temporal features directly from sequence-level patches. We extensively evaluate our approach on two cataract surgery video datasets, Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-07-20T17:57:04Z)
• LoViT: Long Video Transformer for Surgical Phase Recognition [59.06812739441785]
  We present a two-stage method, called Long Video Transformer (LoViT) for fusing short- and long-term temporal information. Our approach outperforms state-of-the-art methods on the Cholec80 and AutoLaparo datasets consistently.
  arXiv  Detail & Related papers  (2023-05-15T20:06:14Z)
• FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment [93.09267863425492]
  We argue that understanding both high-level semantics and internal temporal structures of actions in competitive sports videos is the key to making predictions accurate and interpretable. We construct a new fine-grained dataset, called FineDiving, developed on diverse diving events with detailed annotations on action procedures.
  arXiv  Detail & Related papers  (2022-04-07T17:59:32Z)
• Bottom-Up Temporal Action Localization with Mutual Regularization [107.39785866001868]
  State-of-the-art solutions for TAL involve evaluating the frame-level probabilities of three action-indicating phases. We introduce two regularization terms to mutually regularize the learning procedure. Experiments are performed on two popular TAL datasets, THUMOS14 and ActivityNet1.3.
  arXiv  Detail & Related papers  (2020-02-18T03:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.