Knowledge Distillation for Automated AI Tutor Evaluation
- URL: http://arxiv.org/abs/2607.10647v1
- Date: Sun, 12 Jul 2026 08:28:11 GMT
- Title: Knowledge Distillation for Automated AI Tutor Evaluation
- Abstract summary: We introduce FATE (FLC AI Tutor Evaluator), a specialized language model designed to evaluate AI tutors.<n>Aligned with the four core evaluation tracks from the BEA 2025 Shared Task, our model assesses pedagogical ability across Mistake Identification, Mistake Location, Guidance, and Actionability.<n>We demonstrate FATE's utility as an automated evaluator by benchmarking instructional responses generated by popular commercial models, including ChatGPT, Claude, Gemini, and DeepSeek.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The rapid integration of Large Language Models (LLMs) into K-12 and higher education has outpaced the development of reliable methods for evaluating their pedagogical quality. As the research community starts to explore the space of automating evaluation of AI tutors, we introduce FATE (FLC AI Tutor Evaluator), a specialized 8B-parameter language model designed to evaluate AI tutors. Aligned with the four core evaluation tracks from the BEA 2025 Shared Task, our model assesses pedagogical ability across Mistake Identification, Mistake Location, Guidance, and Actionability. Because pedagogical evaluation is a specialized task with limited labeled data, we leverage knowledge distillation from a frontier LLM to generate additional supervision, yielding absolute performance gains up to 22.63 percentage points. Finally, we demonstrate FATE's utility as an automated evaluator by benchmarking instructional responses generated by popular commercial models, including ChatGPT, Claude, Gemini, and DeepSeek. On average, we have found that Gemini 2.5 Flash perfomed best (82.88%), then ChatGPT 5.5 Instant (80.75%), followed by DeepSeek V4 Flash (80.13%) and Claude Sonnet 4.6 (74.00%).
Related papers
- EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models [66.28519000191191]
EmoSBench is the first comprehensive EI evaluation benchmark for Spoken Language Models.<n>EmoS reaches 83.8% accuracy, approaching human-level performance.
arXiv Detail & Related papers (2026-08-10T06:58:30Z) - EduGuard: A Safe RAG-Based LLM Tutor for Programming Education [1.2999518604217852]
Generative AI (GenAI) is increasingly used by students for programming explanation, debug, and assignment support.<n>This paper presents EduGuard, a safe retrieval-augmented generation (RAG) tutoring framework for introductory programming.<n>EduGuard integrates query understanding, instructor-approved course retrieval, pedagogical strategy selection, rubric-aware generation, claim-level verification, and overreliance control.
arXiv Detail & Related papers (2026-07-17T08:18:18Z) - PersonalAI 2.0: Enhancing knowledge graph traversal/retrieval with planning mechanism for Personalized LLM Agents [53.369140518949656]
We introduce PersonalAI 2.0 (PAI-2), a novel framework to enhance large language model (LLM) based systems through integration of external knowledge graphs (KG)<n>The proposed approach addresses key limitations of existing Retrieval-Augmented Generation (GraphRAG) methods by incorporating a dynamic, multistage query processing pipeline.<n>We show that use of graph algorithms (e.g. BeamSearch, WaterCircles) gain superior results compared to standard flatten retriever on average 6%, while enabled search plan enhancement mechanism gain 18% boost compared to disabled one by LLM-as-a-Judge across six datasets.
arXiv Detail & Related papers (2026-05-13T13:06:30Z) - From Domain Understanding to Design Readiness: a playbook for GenAI-supported learning in Software Engineering [0.4737991126491217]
We report a two-week milestone in a master's course where 29 students used a customized ChatGPT tutor to learn cryptocurrency basics.<n>Results suggest that genAI-supported learning can complement instruction in domain understanding and modeling tasks, while leaving room to improve tone and follow-up structure.
arXiv Detail & Related papers (2026-03-31T18:27:42Z) - Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains [97.5573252172065]
We train a family of Automatic Reasoning Evaluators (FARE) with a simple iterative rejection-sampling supervised finetuning approach.<n>FARE-8B challenges larger specialized RL-trained evaluators and FARE-20B sets the new standard for open-source evaluators.<n>As inference-time rerankers, FARE-20B achieves near-oracle performance on MATH.
arXiv Detail & Related papers (2025-10-20T17:52:06Z) - Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors [6.891852148875869]
The task consisted of five tracks designed to automatically evaluate the AI tutor's performance across key dimensions of mistake identification.<n>The best results for the four pedagogical ability assessment tracks range between macro F1 scores of 58.34 (for providing guidance) and 71.81 (for mistake identification) on three-class problems.
arXiv Detail & Related papers (2025-07-11T10:57:36Z) - Exploring Expert Failures Improves LLM Agent Tuning [74.0772570556016]
We propose Exploring Expert Failures (EEF), which identifies beneficial actions from failed expert trajectories.<n>EEF successfully solves some previously unsolvable subtasks and improves agent tuning performance.
arXiv Detail & Related papers (2025-04-17T17:53:54Z) - Do Tutors Learn from Equity Training and Can Generative AI Assess It? [2.116573423199236]
We evaluate tutor performance within an online lesson on enhancing tutors' skills when responding to students in potentially inequitable situations.<n>We find marginally significant learning gains with increases in tutors' self-reported confidence in their knowledge.<n>This work makes available a dataset of lesson log data, tutor responses, rubrics for human annotation, and generative AI prompts.
arXiv Detail & Related papers (2024-12-15T17:36:40Z) - Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors [7.834688858839734]
We investigate whether current state-of-the-art large language models (LLMs) are effective as AI tutors.<n>We propose a unified evaluation taxonomy with eight pedagogical dimensions based on key learning sciences principles.<n>We release MRBench - a new evaluation benchmark containing 192 conversations and 1,596 responses from seven state-of-the-art LLM-based and human tutors.
arXiv Detail & Related papers (2024-12-12T16:24:35Z) - Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark [62.58869921806019]
We propose a task decomposition evaluation framework based on GPT-4o to automatically construct a new training dataset.
We design innovative training strategies to effectively distill GPT-4o's evaluation capabilities into a 7B open-source MLLM, MiniCPM-V-2.6.
Experimental results demonstrate that our distilled open-source MLLM significantly outperforms the current state-of-the-art GPT-4o-base baseline.
arXiv Detail & Related papers (2024-11-23T08:06:06Z) - LGD: Label-guided Self-distillation for Object Detection [59.9972914042281]
We propose the first self-distillation framework for general object detection, termed LGD (Label-Guided self-Distillation)
Our framework involves sparse label-appearance encoding, inter-object relation adaptation and intra-object knowledge mapping to obtain the instructive knowledge.
Compared with a classical teacher-based method FGFI, LGD not only performs better without requiring pretrained teacher but also with 51% lower training cost beyond inherent student learning.
arXiv Detail & Related papers (2021-09-23T16:55:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.