TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students

• URL: http://arxiv.org/abs/2505.01563v1
• Date: Fri, 02 May 2025 20:03:21 GMT
• Title: TutorGym: A Testbed for Evaluating AI Agents as Tutors and Students
• Abstract summary: TutorGym is a standard interface for testing artificial intelligence (AI) agents within existing intelligent tutoring systems (ITS)<n>At each step of problem-solving, AI agents are asked what they would do as a tutor or as a learner.<n>As students, agents directly learn from ITS instruction, and their mistakes and learning trajectories can be compared to student data.
• Score: 0.283600654802951
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent improvements in large language model (LLM) performance on academic benchmarks, such as MATH and GSM8K, have emboldened their use as standalone tutors and as simulations of human learning. However, these new applications require more than evaluations of final solution generation. We introduce TutorGym to evaluate these applications more directly. TutorGym is a standard interface for testing artificial intelligence (AI) agents within existing intelligent tutoring systems (ITS) that have been tested and refined in classroom studies, including Cognitive Tutors (CTAT), Apprentice Tutors, and OATutors. TutorGym is more than a simple problem-solution benchmark, it situates AI agents within the interactive interfaces of existing ITSs. At each step of problem-solving, AI agents are asked what they would do as a tutor or as a learner. As tutors, AI agents are prompted to provide tutoring support -- such as generating examples, hints, and step-level correctness feedback -- which can be evaluated directly against the adaptive step-by-step support provided by existing ITSs. As students, agents directly learn from ITS instruction, and their mistakes and learning trajectories can be compared to student data. TutorGym establishes a common framework for training and evaluating diverse AI agents, including LLMs, computational models of learning, and reinforcement learning agents, within a growing suite of learning environments. Currently, TutorGym includes 223 different tutor domains. In an initial evaluation, we find that current LLMs are poor at tutoring -- none did better than chance at labeling incorrect actions, and next-step actions were correct only ~52-70% of the time -- but they could produce remarkably human-like learning curves when trained as students with in-context learning.

Related papers

• EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners [1.6405027086146922]
  tutoring is long-horizon, since a learner improves over days and weeks rather than in a single turn.<n>We introduce EduClaw-Bench, a benchmark that places an agent tutor in a continuous 30-day relationship with a simulated learner.
  arXiv  Detail & Related papers  (2026-08-04T06:47:26Z)
• Regulating the AI Tutor: Intentions, Help-Seeking, and Self-Regulated Learning in Adolescent GenAI Use [2.024558551027881]
  This work-in-progress examines open-ended conversational data from 98 Grade-9 students across three German Gymnasium schools.<n>We collected pre-post domain knowledge, pre-chat learning needs, and self-reported cognitive load.<n>Although students overwhelmingly selected scaffolded support before the chat, their interactions were dominated by instrumental requests.
  arXiv  Detail & Related papers  (2026-06-07T10:47:52Z)
• Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflows [48.61619205237941]
  EduAgentBench is a source-grounded benchmark for evaluating tutor agents across the full scope of teaching work.<n>It contains 150 quality-controlled tasks across three capability surfaces: professional pedagogical judgment, situated multi-turn tutoring, and Canvas-style teaching workflow completion.<n>To our knowledge, EduAgentBench is the first theory-grounded and realistic benchmark for evaluating the holistic teaching capability of tutor agents.
  arXiv  Detail & Related papers  (2026-05-14T03:34:25Z)
• Toward Training Superintelligent Software Agents through Self-Play SWE-RL [66.11447353341926]
  Self-play SWE-RL is a first step toward training paradigms for superintelligent software agents.<n>Our approach takes minimal data assumptions, only requiring access to sandboxed repositories with source code and installed dependencies.<n>Our results, albeit early, suggest a path where agents autonomously gather extensive learning experiences from real-world software repositories.
  arXiv  Detail & Related papers  (2025-12-21T00:49:40Z)
• An Experience Report on a Pedagogically Controlled, Curriculum-Constrained AI Tutor for SE Education [4.976713294177978]
  This paper presents the design and pilot evaluation of RockStartIT Tutor, an AI-powered assistant developed for a digital programming and computational thinking course within the RockStartIT initiative.<n> Powered by GPT-4 via OpenAI's Assistant API, the tutor employs a novel prompting strategy and a modular, semantically tagged knowledge base to deliver context-aware, personalized, and curriculum-constrained support for secondary school students.
  arXiv  Detail & Related papers  (2025-12-08T12:54:37Z)
• Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning [84.70211451226835]
  Large Language Model (LLM) Agents are constrained by a dependency on human-curated data.<n>We introduce Agent0, a fully autonomous framework that evolves high-performing agents without external data.<n>Agent0 substantially boosts reasoning capabilities, improving the Qwen3-8B-Base model by 18% on mathematical reasoning and 24% on general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-11-20T05:01:57Z)
• CATArena: Evaluation of LLM Agents through Iterative Tournament Competitions [49.02422075498554]
  Large Language Model (LLM) agents have evolved from basic text generation to autonomously completing complex tasks through interaction with external tools.<n>In this work, we emphasize the importance of learning ability, including both self-improvement and peer-learning, as a core driver for agent evolution toward human-level intelligence.<n>We propose an iterative, competitive peer-learning framework, which allows agents to refine and optimize their strategies through repeated interactions and feedback.
  arXiv  Detail & Related papers  (2025-10-30T15:22:53Z)
• SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience [71.82719117238307]
  We propose SEAgent, an agentic self-evolving framework enabling computer-use agents to evolve through interactions with unfamiliar software.<n>We validate the effectiveness of SEAgent across five novel software environments within OS-World.<n>Our approach achieves a significant improvement of 23.2% in success rate, from 11.3% to 34.5%, over a competitive open-source CUA.
  arXiv  Detail & Related papers  (2025-08-06T17:58:46Z)
• InqEduAgent: Adaptive AI Learning Partners with Gaussian Process Augmentation [4.96669107440958]
  This paper proposes an LLM-empowered agent model for simulating and selecting learning partners tailored to inquiry-oriented learning.<n>Generative agents are designed to capture cognitive and evaluative features of learners in real-world scenarios.<n>The experimental results show the optimal performance of InqEduAgent in most knowledge-learning scenarios and LLM environment.
  arXiv  Detail & Related papers  (2025-08-05T07:33:48Z)
• Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidance [58.21767225794469]
  Large language model (LLM) agents often struggle in environments where rules and required domain knowledge frequently change.<n>We propose the Adaptive Reflective Interactive Agent (ARIA) to continuously learn updated domain knowledge at test time.<n>ARIA is deployed within TikTok Pay serving over 150 million monthly active users.
  arXiv  Detail & Related papers  (2025-07-23T02:12:32Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [66.1850490474361]
  We conduct the first academic study to explore developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants, GitHub Copilot and OpenHands.<n>Our results show agents have the potential to assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.<n>Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes.<n>We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• Do Tutors Learn from Equity Training and Can Generative AI Assess It? [2.116573423199236]
  We evaluate tutor performance within an online lesson on enhancing tutors' skills when responding to students in potentially inequitable situations.<n>We find marginally significant learning gains with increases in tutors' self-reported confidence in their knowledge.<n>This work makes available a dataset of lesson log data, tutor responses, rubrics for human annotation, and generative AI prompts.
  arXiv  Detail & Related papers  (2024-12-15T17:36:40Z)
• AI2T: Building Trustable AI Tutors by Interactively Teaching a Self-Aware Learning Agent [2.0937431058291933]
  Authors tutor AI2T by providing a few step-by-step solutions and then grading AI2T's own problem-solving attempts.<n>From just 20-30 minutes of interactive training, AI2T can induce robust rules for step-by-step solution tracking.
  arXiv  Detail & Related papers  (2024-11-26T22:39:11Z)
• ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning [78.42927884000673]
  ExACT is an approach to combine test-time search and self-learning to build o1-like models for agentic applications.<n>We first introduce Reflective Monte Carlo Tree Search (R-MCTS), a novel test time algorithm designed to enhance AI agents' ability to explore decision space on the fly.<n>Next, we introduce Exploratory Learning, a novel learning strategy to teach agents to search at inference time without relying on any external search algorithms.
  arXiv  Detail & Related papers  (2024-10-02T21:42:35Z)
• AI-Tutoring in Software Engineering Education [0.7631288333466648]
  We conducted an exploratory case study by integrating the GPT-3.5-Turbo model as an AI-Tutor within the APAS Artemis. The findings highlight advantages, such as timely feedback and scalability. However, challenges like generic responses and students' concerns about a learning progress inhibition when using the AI-Tutor were also evident.
  arXiv  Detail & Related papers  (2024-04-03T08:15:08Z)
• Impact of Decentralized Learning on Player Utilities in Stackelberg Games [57.08270857260131]
  In many two-agent systems, each agent learns separately and the rewards of the two agents are not perfectly aligned. We model these systems as Stackelberg games with decentralized learning and show that standard regret benchmarks result in worst-case linear regret for at least one player. We develop algorithms to achieve near-optimal $O(T2/3)$ regret for both players with respect to these benchmarks.
  arXiv  Detail & Related papers  (2024-02-29T23:38:28Z)
• Evaluating Large Language Models on the GMAT: Implications for the Future of Business Education [0.13654846342364302]
  This study introduces the first benchmark to assess the performance of seven major Large Language Models (LLMs) Our analysis shows that most LLMs outperform human candidates, with GPT-4 Turbo not only outperforming the other models but also surpassing the average scores of graduate students at top business schools. While AI's promise in education, assessment, and tutoring is clear, challenges remain.
  arXiv  Detail & Related papers  (2024-01-02T03:54:50Z)
• Experiential Co-Learning of Software-Developing Agents [83.34027623428096]
  Large language models (LLMs) have brought significant changes to various domains, especially in software development. We introduce Experiential Co-Learning, a novel LLM-agent learning framework. Experiments demonstrate that the framework enables agents to tackle unseen software-developing tasks more effectively.
  arXiv  Detail & Related papers  (2023-12-28T13:50:42Z)
• Modelling Assessment Rubrics through Bayesian Networks: a Pragmatic Approach [40.06500618820166]
  This paper presents an approach to deriving a learner model directly from an assessment rubric. We illustrate how the approach can be applied to automatize the human assessment of an activity developed for testing computational thinking skills.
  arXiv  Detail & Related papers  (2022-09-07T10:09:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.