EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners

• URL: http://arxiv.org/abs/2608.03206v1
• Date: Tue, 04 Aug 2026 06:47:26 GMT
• Title: EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners
• Abstract summary: tutoring is long-horizon, since a learner improves over days and weeks rather than in a single turn.<n>We introduce EduClaw-Bench, a benchmark that places an agent tutor in a continuous 30-day relationship with a simulated learner.
• Score: 1.6405027086146922
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) power educational applications from tutoring to essay scoring, but each is a point solution to a single task, and only recently have these point solutions been integrated into agents operating over a learning management system (LMS). Yet tutoring is long-horizon, since a learner improves over days and weeks rather than in a single turn, and no benchmark evaluates an agent tutor across a sustained relationship. We introduce EduClaw-Bench, a benchmark that places an agent tutor in a continuous 30-day relationship with a simulated learner grounded in knowledge tracing (KT), whose knowledge-concept mastery, from a KT model trained on real-student data, drives its answers and is probed for learning gain across 55 scenarios. Each agent is scored on three primary axes (learning gain, responsiveness, and helpfulness) and two curriculum-design axes (Gagné and Rosenshine), with helpfulness and the curriculum axes judged by a cross-family panel of three LLM judges. Evaluating 10 agent adapters over three base-model tiers yields two findings that single-tier, single-session evaluation cannot reach. First, tutoring quality belongs to the base model and the agent harness together rather than either alone. Second, almost no combination sustains good tutoring over the full horizon. A calibration check ($\text{ECE}=0.049$) and a live-classroom field study confirm that the simulated learner and its measurements track reality. Our work is a step toward trustworthy AI tutors for future education.

Related papers

• Sensi: Learn One Thing at a Time -- Curriculum-Based Test-Time Learning for LLM Game Agents [0.0]
  We present Sensi, an agent architecture for ARC-AGI-3 game-playing challenge that introduces structured test-time learning.<n>We report results across two iterations: Sensi v1 solves 2 game levels using the two-player architecture alone, while Sensi v2 adds curriculum learning and solves 0 levels.
  arXiv  Detail & Related papers  (2026-03-18T12:59:26Z)
• IntelliCode: A Multi-Agent LLM Tutoring System with Centralized Learner Modeling [2.0467354053171243]
  We introduce IntelliCode, a multi-agent LLM tutoring system built around a centralized, versioned learner state.<n>A StateGraph Orchestrator coordinates six specialized agents: skill assessment, learner profiling, graduated hinting, curriculum selection, spaced repetition, and engagement monitoring.<n>This architecture enables auditable mastery updates, proficiency-aware hints, dependency-aware curriculum adaptation, and safety-aligned prompting.
  arXiv  Detail & Related papers  (2025-12-21T10:07:06Z)
• Multi-Agent Evolve: LLM Self-Improve through Co-evolution [53.00458074754831]
  Reinforcement Learning (RL) has demonstrated significant potential in enhancing the reasoning capabilities of large language models (LLMs)<n>Recent Self-Play RL methods, inspired by the success of the paradigm in games and Go, aim to enhance LLM reasoning capabilities without human-annotated data.<n>We propose Multi-Agent Evolve (MAE), a framework that enables LLMs to self-evolve in solving diverse tasks, including mathematics, reasoning, and general knowledge Q&A.
  arXiv  Detail & Related papers  (2025-10-27T17:58:02Z)
• AdaSwitch: Adaptive Switching Generation for Knowledge Distillation [58.647880811071495]
  Small language models (SLMs) are crucial for applications with strict latency and computational constraints.<n>We propose AdaSwitch, a novel approach that combines on-policy and off-policy generation at the token level.<n>AdaSwitch consistently improves accuracy, offering a practical and effective method for distilling SLMs with acceptable additional overhead.
  arXiv  Detail & Related papers  (2025-10-09T06:38:37Z)
• TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language Models [10.963195858672627]
  TutorBench is a dataset and evaluation benchmark designed to rigorously evaluate the core tutoring skills of large language models (LLMs)<n>Samples are drawn from three common tutoring tasks: (i) generating adaptive explanations tailored to a student's confusion, (ii) providing actionable feedback on a student's work, and (iii) promoting active learning through effective hint generation.<n>We evaluate 16 frontier LLMs on TutorBench and present a detailed analysis of their performance and behavior.
  arXiv  Detail & Related papers  (2025-10-03T01:41:09Z)
• Self-Challenging Language Model Agents [98.62637336505242]
  We propose the Self-Challenging framework for training an agent on high-quality tasks that are generated by itself.<n>The framework achieves over a two-fold improvement in Llama-3.1-8B-Instruct, despite using only self-generated training data.
  arXiv  Detail & Related papers  (2025-06-02T14:23:33Z)
• Is your multimodal large language model a good science tutor? [14.505855717011725]
  Multimodal large language models (MLLMs) demonstrate impressive performance on scientific reasoning tasks.<n>We propose a framework that evaluates MLLMs as science tutors using a comprehensive educational rubric and a simulated student model.
  arXiv  Detail & Related papers  (2025-05-09T20:38:23Z)
• TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students [0.283600654802951]
  TutorGym is a standard interface for testing artificial intelligence (AI) agents within existing intelligent tutoring systems (ITS)<n>At each step of problem-solving, AI agents are asked what they would do as a tutor or as a learner.<n>As students, agents directly learn from ITS instruction, and their mistakes and learning trajectories can be compared to student data.
  arXiv  Detail & Related papers  (2025-05-02T20:03:21Z)
• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference.<n>Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• Learning to Use Tools via Cooperative and Interactive Agents [58.77710337157665]
  Tool learning empowers large language models (LLMs) as agents to use external tools and extend their utility. We propose ConAgents, a Cooperative and interactive Agents framework, which coordinates three specialized agents for tool selection, tool execution, and action calibration separately. Our experiments on three datasets show that the LLMs, when equipped with ConAgents, outperform baselines with substantial improvement.
  arXiv  Detail & Related papers  (2024-03-05T15:08:16Z)
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark [81.42376626294812]
  We present Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs. We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision.
  arXiv  Detail & Related papers  (2023-06-11T14:01:17Z)
• MERMAIDE: Learning to Align Learners using Model-Based Meta-Learning [62.065503126104126]
  We study how a principal can efficiently and effectively intervene on the rewards of a previously unseen learning agent in order to induce desirable outcomes. This is relevant to many real-world settings like auctions or taxation, where the principal may not know the learning behavior nor the rewards of real people. We introduce MERMAIDE, a model-based meta-learning framework to train a principal that can quickly adapt to out-of-distribution agents.
  arXiv  Detail & Related papers  (2023-04-10T15:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.