Evaluating and Improving Pedagogical Fit in LLM-Based AI Tutors with the Pedagogical Suitability Index

• URL: http://arxiv.org/abs/2608.05411v1
• Date: Wed, 05 Aug 2026 21:07:10 GMT
• Title: Evaluating and Improving Pedagogical Fit in LLM-Based AI Tutors with the Pedagogical Suitability Index
• Abstract summary: The Pedagogical Suitability Index (PSI) is a composite metric of six theory-informed sub-scores.<n>We evaluate four large language models (LLMs) across 240 scenario-based evaluations.<n>PSI-guided feedback substantially improved weak-performing cases.
• Score: 0.26508608365976566
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used as AI tutors, but a correct answer is not always a pedagogically appropriate one. In classroom learning, effective help depends not only on correctness, but also on whether a response matches the learner's current foundation, the course sequence, and the timing of concept introduction. Existing evaluations focus mainly on answer quality, leaving this instructional fit under-measured. We present the Pedagogical Suitability Index (PSI), a composite metric of six theory-informed sub-scores that evaluates how well LLM-generated tutoring responses align with learner readiness and curricular progression, and we further use PSI as a structured feedback signal for response improvement. We evaluate four LLM tutors (ChatGPT, Gemini, Gemma4, and Qwen3) across 240 scenario-based evaluations using paired standard and defective prompts, then apply a PSI-guided regeneration protocol to 62 weak-performing cases. Baseline differences across the four tested models were modest overall (PSI range: 0.557 to 0.638), and open-weight and closed models did not exhibit a clear separation in pedagogical fit. Under the tested prompt perturbations, overall PSI remained largely stable (Delta = -0.002), though sub-score trade-offs emerged. More importantly, PSI-guided feedback substantially improved weak-performing cases: 51 of 62 cases improved (82.3%). Focused manual evaluation of the 62 PSI-selected weak cases provides initial evidence that the identified weaknesses are instructionally meaningful and that many PSI-guided regenerations correspond to human-judged improvement. These results suggest that learner- and curriculum-aware alignment may matter more for effective tutoring than model category alone, and that such alignment is both measurable and improvable.

Related papers

• How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks [60.853503196898906]
  Low reported scores on leading physics benchmarks suggest frontier language models struggle with advanced physics.<n>We revisit these reported findings by evaluating frontier models on six widely used physics benchmarks and auditing them with experts.
  arXiv  Detail & Related papers  (2026-09-11T16:06:50Z)
• EduGuard: A Safe RAG-Based LLM Tutor for Programming Education [1.2999518604217852]
  Generative AI (GenAI) is increasingly used by students for programming explanation, debug, and assignment support.<n>This paper presents EduGuard, a safe retrieval-augmented generation (RAG) tutoring framework for introductory programming.<n>EduGuard integrates query understanding, instructor-approved course retrieval, pedagogical strategy selection, rubric-aware generation, claim-level verification, and overreliance control.
  arXiv  Detail & Related papers  (2026-07-17T08:18:18Z)
• Measuring Whether LLM Tutors Teach or Solve: A Diagnostic for Educational Impact [4.6840536642016195]
  We study whether public tutoring benchmarks distinguish learning-supportive behavior from mere answer production.<n>We propose a lightweight diagnostic rubric based on the gap between solving-oriented and pedagogy-oriented benchmark performance.
  arXiv  Detail & Related papers  (2026-06-15T04:32:18Z)
• Mitigating LLM biases toward spurious social contexts using direct preference optimization [5.169989177779799]
  We investigate model robustness to spurious social contexts using the largest publicly available dataset of U.S. classroom transcripts.<n>We find that irrelevant contextual information can shift model predictions by up to 1.48 points on a 7-point scale.<n>We propose **Debiasing-DPO**, a self-supervised training method that pairs neutral reasoning generated from the query alone with the model's biased reasoning.
  arXiv  Detail & Related papers  (2026-04-02T23:42:20Z)
• ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization [73.0780809974414]
  We propose a Reflective Autoformalization method that integrates semantic consistency evaluation into the autoformalization process.<n>This enables the model to iteratively generate formal statements, assess its semantic fidelity, and self-correct identified errors.<n>Experiments show that ReForm achieves an average improvement of 22.6 percentage points over the strongest baselines.
  arXiv  Detail & Related papers  (2025-10-28T16:22:54Z)
• Bridging Supervised Learning and Reinforcement Learning in Math Reasoning [55.889740979706815]
  Reinforcement Learning (RL) has played a central role in the recent surge of math abilities by enabling self-improvement through binary verifier signals.<n>In this work, we propose Negative-aware Fine-Tuning (NFT) -- a supervised approach that enables LLMs to reflect on their failures and improve autonomously with no external teachers.
  arXiv  Detail & Related papers  (2025-05-23T17:17:40Z)
• Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study [0.0]
  Large Language Models (LLMs) hold promise as dynamic instructional aids.<n>Yet, it remains unclear whether LLMs can replicate the adaptivity of intelligent tutoring systems (ITS)
  arXiv  Detail & Related papers  (2025-04-07T23:57:32Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Education [41.69192181482715]
  We present the development of GuideLM, a fine-tuned large language model (LLMs) for programming education.<n>GuideLM has been integrated into the C Compiler (DCC), an educational C compiler that leverages LLMs to generate pedagogically sound error explanations.<n>We conducted an expert analysis of 400 responses per model, comparing their pedagogical effectiveness against base OpenAI models.<n>Results indicate that GuideLM and GuideLM-mini improve pedagogical performance, with an 8% increase in Socratic guidance and a 58% improvement in economy of words compared to GPT-4o.
  arXiv  Detail & Related papers  (2025-02-27T21:23:56Z)
• Self-Taught Evaluators [77.92610887220594]
  We present an approach that aims to im-proves without human annotations, using synthetic training data only. Our Self-Taught Evaluator can improve a strong LLM from 75.4 to 88.3 on RewardBench.
  arXiv  Detail & Related papers  (2024-08-05T17:57:02Z)
• Improving the Validity of Automatically Generated Feedback via Reinforcement Learning [46.667783153759636]
  We propose a framework for feedback generation that optimize both correctness and alignment using reinforcement learning (RL)<n>Specifically, we use GPT-4's annotations to create preferences over feedback pairs in an augmented dataset for training via direct preference optimization (DPO)
  arXiv  Detail & Related papers  (2024-03-02T20:25:50Z)
• LIMA: Less Is More for Alignment [112.93890201395477]
  We train LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases.
  arXiv  Detail & Related papers  (2023-05-18T17:45:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.