EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibration

• URL: http://arxiv.org/abs/2607.18529v1
• Date: Mon, 20 Jul 2026 21:45:00 GMT
• Title: EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibration
• Abstract summary: EduPanel is a rubric-grounded, learner-conditioned LLM judge that decomposes evaluation across specialized agents.<n>It achieves reliability comparable to a median human expert across expert studies, architecture ablations, and learner-persona analyses.<n>These results suggest that EduPanel can serve as effective assistants for educational evaluation rather than replacements for human experts.
• Score: 49.59690207400984
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Teaching videos are becoming a major medium for education, creating a growing need for scalable evaluation of their pedagogical quality. Existing automatic judges do not fully address this setting because teaching quality depends on multimodal evidence and should be evaluated with respect to the intended learner rather than as a universal property. We present EduPanel, a rubric-grounded, learner-conditioned LLM judge that decomposes evaluation across specialized agents to produce interpretable assessments for different aspects of teaching quality. Across expert studies, architecture ablations, and learner-persona analyses, EduPanel achieves reliability comparable to a median human expert. In expert evaluation, its feedback improves scoring accuracy (MAE 0.87 to 0.73), while experts remain able to detect unreliable outputs (AUC = 0.77) instead of accepting them blindly. These results suggest that EduPanel can serve as effective assistants for educational evaluation rather than replacements for human experts.

Related papers

• Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflows [48.61619205237941]
  EduAgentBench is a source-grounded benchmark for evaluating tutor agents across the full scope of teaching work.<n>It contains 150 quality-controlled tasks across three capability surfaces: professional pedagogical judgment, situated multi-turn tutoring, and Canvas-style teaching workflow completion.<n>To our knowledge, EduAgentBench is the first theory-grounded and realistic benchmark for evaluating the holistic teaching capability of tutor agents.
  arXiv  Detail & Related papers  (2026-05-14T03:34:25Z)
• An Explainable AI Assistant for Introductory Programming Education: Improving Feedback Reliability with Instructor-AI Collaboration [6.172982108802445]
  We present an AI-driven classroom assistant that leverages an explainable AI model to analyze student code, map logical errors to instructor-identified misconceptions, and deliver instructor-authored feedback.<n>Results indicate that the assistant can provide accurate, instructor-verified feedback to students while fostering a positive experience.
  arXiv  Detail & Related papers  (2026-05-12T06:35:55Z)
• Scaling Equitable Reflection Assessment in Education via Large Language Models and Role-Based Feedback Agents [2.825140278227664]
  Formative feedback is one of the most effective drivers of student learning.<n>In large or low-resource courses, instructors often lack the time, staffing, and bandwidth required to review and respond to every student reflection.<n>This paper presents a theory-grounded system that uses five coordinated role-based LLM agents to score learner reflections.
  arXiv  Detail & Related papers  (2025-11-14T09:46:21Z)
• AI Judges in Design: Statistical Perspectives on Achieving Human Expert Equivalence With Vision-Language Models [3.092385483349516]
  This paper introduces a rigorous statistical framework to determine whether an AI judge's ratings match those of human experts.<n>We apply this framework in a case study evaluating four VLM-based judges on key design metrics.<n>Results show that the top-performing AI judge achieves expert-level agreement for uniqueness and drawing quality.
  arXiv  Detail & Related papers  (2025-04-01T16:20:29Z)
• MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors [82.91830877219822]
  We present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation.<n>MathTutorBench contains datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching.<n>We evaluate a wide set of closed- and open-weight models and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching.
  arXiv  Detail & Related papers  (2025-02-26T08:43:47Z)
• Intelligent Tutors Beyond K-12: An Observational Study of Adult Learner Engagement and Academic Impact [0.0]
  This study examines the adoption, usage patterns, and effectiveness of a novel tutoring system, Apprentice Tutors, among adult learners at a state technical college.<n>Our findings reveal key temporal patterns in tutor engagement and provide evidence of learning within tutors, as determined through skill improvement in knowledge components across tutors.<n>These results suggest that intelligent tutors are a viable tool for adult learners, warranting further research into their long-term impact on this population.
  arXiv  Detail & Related papers  (2025-02-23T15:36:22Z)
• HREF: Human Response-Guided Evaluation of Instruction Following in Language Models [61.273153125847166]
  We develop a new evaluation benchmark, Human Response-Guided Evaluation of Instruction Following (HREF)<n>In addition to providing reliable evaluation, HREF emphasizes individual task performance and is free from contamination.<n>We study the impact of key design choices in HREF, including the size of the evaluation set, the judge model, the baseline model, and the prompt template.
  arXiv  Detail & Related papers  (2024-12-20T03:26:47Z)
• Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors [7.834688858839734]
  We investigate whether current state-of-the-art large language models (LLMs) are effective as AI tutors.<n>We propose a unified evaluation taxonomy with eight pedagogical dimensions based on key learning sciences principles.<n>We release MRBench - a new evaluation benchmark containing 192 conversations and 1,596 responses from seven state-of-the-art LLM-based and human tutors.
  arXiv  Detail & Related papers  (2024-12-12T16:24:35Z)
• ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models [53.00812898384698]
  We argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking. We highlight how cognitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such as Likert. We propose the ConSiDERS-The-Human evaluation framework consisting of 6 pillars -- Consistency, Scoring Criteria, Differentiating, User Experience, Responsible, and Scalability.
  arXiv  Detail & Related papers  (2024-05-28T22:45:28Z)
• Evaluating and Optimizing Educational Content with Large Language Model Judgments [52.33701672559594]
  We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
  arXiv  Detail & Related papers  (2024-03-05T09:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.