A Multi-Agent LLM Framework for Rating the Quality of Surgical Feedback

• URL: http://arxiv.org/abs/2605.25440v1
• Date: Mon, 25 May 2026 05:31:44 GMT
• Title: A Multi-Agent LLM Framework for Rating the Quality of Surgical Feedback
• Abstract summary: Verbal feedback delivered by attending surgeons in the operating room plays a critical formative role in resident skill acquisition.<n>We introduce a two-stage LLM-based framework that discovers interpretable feedback quality criteria grounded in the context of surgical training.<n>Our method uses multi-agent and surgical domain knowledge injection to discover a small set of human interpretable scoring criteria.
• Score: 52.053200111780626
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Verbal feedback delivered by attending surgeons in the operating room plays a critical formative role in resident trainee skill acquisition. Yet, assessing the quality of trainer feedback and its effectiveness in influencing trainee behavior during live surgery remains a challenge. Prior studies assessed feedback content relying on extensive manual annotation by expert human raters and focused on developing broad taxonomies that overlook the qualitative aspects of feedback delivery such as clarity or urgency. Limited existing automated methods, including keyword analysis and topic modeling, also fail to capture these nuanced aspects. We introduce a two-stage LLM-based framework that discovers interpretable feedback quality criteria grounded in the context of surgical training. Our method uses multi-agent prompting and surgical domain knowledge injection to discover a small set of human interpretable scoring criteria (e.g., Encouraging, Urgent, Clear). These criteria are then used to automatically score live surgical feedback via an LLM-as-a-judge approach. Evaluation on 4.2k trainer feedback instances demonstrates that our AI-discovered criteria outperform prior content-based frameworks in predicting feedback effectiveness, including observed trainee behavioral adjustments and trainer approval. This work advances scalable, human-aligned assessment of communication quality in the operating room and provides a foundation for improving surgical teaching practices.

Related papers

• Toward Accessible Psychotherapy Training Using AI-Driven Interactive Patient Avatars [0.45671221781968324]
  Training psychotherapists in Acceptance and Commitment Therapy (ACT) requires repeated practice with meaningful feedback.<n>We introduce a system designed to support ACT-oriented psychotherapy training through spoken dialogue with an embodied virtual patient.
  arXiv  Detail & Related papers  (2026-06-16T11:02:00Z)
• AI for Quality Assurance in the Operating Room [7.402717868707822]
  AI for quality assurance should be understood as a set of tools for augmenting the surgical team, scaling expert review, and helping surgery evolve toward a learning system.<n>This chapter introduces AI-enabled Surgical Quality Assurance as a frame-work for using surgical data to support continuous assessment and improvement in the operating room.
  arXiv  Detail & Related papers  (2026-06-16T09:05:18Z)
• AI-Driven Evaluation of Surgical Skill via Action Recognition [4.92174988745803]
  We propose an AI-driven framework for the automated assessment of microanastomosis performance.<n>Performance is evaluated along five aspects of microanastomosis skill, including overall action execution, motion quality during procedure-critical actions, and general instrument handling.
  arXiv  Detail & Related papers  (2025-12-30T18:45:34Z)
• Explainable AI for Automated User-specific Feedback in Surgical Skill Acquisition [38.38538970682482]
  We examine the effectiveness of explainable AI (XAI)-generated feedback in surgical training through a human-AI study.<n>We compare the impact of XAI-guided feedback against traditional video-based coaching on task outcomes, cognitive load, and trainees' perceptions of AI-assisted learning.
  arXiv  Detail & Related papers  (2025-08-04T16:48:44Z)
• AutoMedEval: Harnessing Language Models for Automatic Medical Capability Evaluation [55.2739790399209]
  We present AutoMedEval, an open-sourced automatic evaluation model with 13B parameters specifically engineered to measure the question-answering proficiency of medical LLMs.<n>The overarching objective of AutoMedEval is to assess the quality of responses produced by diverse models, aspiring to significantly reduce the dependence on human evaluation.
  arXiv  Detail & Related papers  (2025-05-17T07:44:54Z)
• Automating Feedback Analysis in Surgical Training: Detection, Categorization, and Assessment [65.70317151363204]
  This work introduces the first framework for reconstructing surgical dialogue from unstructured real-world recordings.<n>In surgical training, the formative verbal feedback that trainers provide to trainees during live surgeries is crucial for ensuring safety, correcting behavior immediately, and facilitating long-term skill acquisition.<n>Our framework integrates voice activity detection, speaker diarization, and automated speech recaognition, with a novel enhancement that removes hallucinations.
  arXiv  Detail & Related papers  (2024-12-01T10:35:12Z)
• Multi-Modal Self-Supervised Learning for Surgical Feedback Effectiveness Assessment [66.6041949490137]
  We propose a method that integrates information from transcribed verbal feedback and corresponding surgical video to predict feedback effectiveness. Our findings show that both transcribed feedback and surgical video are individually predictive of trainee behavior changes. Our results demonstrate the potential of multi-modal learning to advance the automated assessment of surgical feedback.
  arXiv  Detail & Related papers  (2024-11-17T00:13:00Z)
• Deep Multimodal Fusion for Surgical Feedback Classification [70.53297887843802]
  We leverage a clinically-validated five-category classification of surgical feedback. We then develop a multi-label machine learning model to classify these five categories of surgical feedback from inputs of text, audio, and video modalities. The ultimate goal of our work is to help automate the annotation of real-time contextual surgical feedback at scale.
  arXiv  Detail & Related papers  (2023-12-06T01:59:47Z)
• Video-based Formative and Summative Assessment of Surgical Tasks using Deep Learning [0.8612287536028312]
  We propose a deep learning (DL) model that can automatically and objectively provide a high-stakes summative assessment of surgical skill execution. Formative assessment is generated using heatmaps of visual features that correlate with surgical performance.
  arXiv  Detail & Related papers  (2022-03-17T20:07:48Z)
• Opportunities of a Machine Learning-based Decision Support System for Stroke Rehabilitation Assessment [64.52563354823711]
  Rehabilitation assessment is critical to determine an adequate intervention for a patient. Current practices of assessment mainly rely on therapist's experience, and assessment is infrequently executed due to the limited availability of a therapist. We developed an intelligent decision support system that can identify salient features of assessment using reinforcement learning.
  arXiv  Detail & Related papers  (2020-02-27T17:04:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.