From Correctness to Comprehension: AI Agents for Personalized Error Diagnosis in Education

• URL: http://arxiv.org/abs/2502.13789v1
• Date: Wed, 19 Feb 2025 14:57:51 GMT
• Title: From Correctness to Comprehension: AI Agents for Personalized Error Diagnosis in Education
• Authors: Yi-Fan Zhang, Hang Li, Dingjie Song, Lichao Sun, Tianlong Xu, Qingsong Wen,
• Abstract summary: Large Language Models (LLMs) have demonstrated impressive mathematical reasoning capabilities, achieving near-perfect performance on benchmarks like GSM8K. However, their application in personalized education remains limited due to an overemphasis on correctness over error diagnosis and feedback generation. We introduce textbfMathCCS, a benchmark designed for systematic error analysis and tailored feedback. Second, we develop a sequential error analysis framework that leverages historical data to track trends and improve diagnostic precision. Third, we propose a multi-agent collaborative framework that combines a Time Series Agent for historical analysis and an MLLM Agent for real-
• Score: 24.970741456147447
• License:
• Abstract: Large Language Models (LLMs), such as GPT-4, have demonstrated impressive mathematical reasoning capabilities, achieving near-perfect performance on benchmarks like GSM8K. However, their application in personalized education remains limited due to an overemphasis on correctness over error diagnosis and feedback generation. Current models fail to provide meaningful insights into the causes of student mistakes, limiting their utility in educational contexts. To address these challenges, we present three key contributions. First, we introduce \textbf{MathCCS} (Mathematical Classification and Constructive Suggestions), a multi-modal benchmark designed for systematic error analysis and tailored feedback. MathCCS includes real-world problems, expert-annotated error categories, and longitudinal student data. Evaluations of state-of-the-art models, including \textit{Qwen2-VL}, \textit{LLaVA-OV}, \textit{Claude-3.5-Sonnet} and \textit{GPT-4o}, reveal that none achieved classification accuracy above 30\% or generated high-quality suggestions (average scores below 4/10), highlighting a significant gap from human-level performance. Second, we develop a sequential error analysis framework that leverages historical data to track trends and improve diagnostic precision. Finally, we propose a multi-agent collaborative framework that combines a Time Series Agent for historical analysis and an MLLM Agent for real-time refinement, enhancing error classification and feedback generation. Together, these contributions provide a robust platform for advancing personalized education, bridging the gap between current AI capabilities and the demands of real-world teaching.

Related papers

• Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Framework [64.83955753606443]
  Math Word Problems serve as a crucial benchmark for evaluating Large Language Models' reasoning abilities. Current error classification methods rely on static and predefined categories. We introduce MWPES-300K, a comprehensive dataset containing 304,865 error samples.
  arXiv  Detail & Related papers  (2025-01-26T16:17:57Z)
• ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection [60.297079601066784]
  We introduce ErrorRadar, the first benchmark designed to assess MLLMs' capabilities in error detection. ErrorRadar evaluates two sub-tasks: error step identification and error categorization. It consists of 2,500 high-quality multimodal K-12 mathematical problems, collected from real-world student interactions. Results indicate significant challenges still remain, as GPT-4o with best performance is still around 10% behind human evaluation.
  arXiv  Detail & Related papers  (2024-10-06T14:59:09Z)
• IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation [15.895295957106772]
  We propose an ID-induced prompt synthesis framework for evaluating Large Language Models (LLMs) Our data synthesis framework prioritizes both breadth and specificity. It can generate prompts that comprehensively evaluate the capabilities of LLMs. We will release a dataset of over 3,000 carefully crafted prompts to facilitate evaluation research of LLMs.
  arXiv  Detail & Related papers  (2024-09-27T16:29:12Z)
• Is Difficulty Calibration All We Need? Towards More Practical Membership Inference Attacks [16.064233621959538]
  We propose a query-efficient and computation-efficient MIA that directly textbfRe-levertextbfAges the original membershitextbfP scores to mtextbfItigate the errors in textbfDifficulty calibration.
  arXiv  Detail & Related papers  (2024-08-31T11:59:42Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Overcoming Pitfalls in Graph Contrastive Learning Evaluation: Toward Comprehensive Benchmarks [60.82579717007963]
  We introduce an enhanced evaluation framework designed to more accurately gauge the effectiveness, consistency, and overall capability of Graph Contrastive Learning (GCL) methods.
  arXiv  Detail & Related papers  (2024-02-24T01:47:56Z)
• Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program [0.0]
  This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks. We show that errors can be predicted with moderate model performance (AUC=0.65-0.75) and that model performance generalizes well across applications. We demonstrate the usefulness of the model in the context of auditing, where prioritizing tasks with high predicted error probabilities considerably increases the amount of corrected annotation errors.
  arXiv  Detail & Related papers  (2023-10-08T21:21:19Z)
• Analysis of the Reasoning with Redundant Information Provided Ability of Large Language Models [0.0]
  Large Language Models (LLMs) have demonstrated impressive capabilities across a range of natural language processing tasks. To address this gap, a new form of Question-Answering (QA) task, termed Reasoning with Redundant Information Provided (RRIP), is introduced. This study evaluates two popular LLMs, LlaMA2-13B-chat and generative pre-trained transformer 3.5 (GPT-3.5), contrasting their performance on traditional QA tasks against RRIP tasks.
  arXiv  Detail & Related papers  (2023-10-06T06:20:06Z)
• Fusing Models with Complementary Expertise [42.099743709292866]
  We consider the Fusion of Experts (FoE) problem of fusing outputs of expert models with complementary knowledge of the data distribution. Our method is applicable to both discriminative and generative tasks. We extend our method to the "frugal" setting where it is desired to reduce the number of expert model evaluations at test time.
  arXiv  Detail & Related papers  (2023-10-02T18:31:35Z)
• A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check [53.152011258252315]
  We show that using phonetic and graphic information reasonably is effective for Chinese Spelling Check. Models are sensitive to the error distribution of the test set, which reflects the shortcomings of models. The commonly used benchmark, SIGHAN, can not reliably evaluate models' performance.
  arXiv  Detail & Related papers  (2023-07-25T17:02:38Z)
• From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models [55.137869702763375]
  This paper explores a novel PLM reuse paradigm, Knowledge Integration (KI) KI aims to merge the knowledge from different teacher-PLMs, each of which specializes in a different classification problem, into a versatile student model. We then design a Model Uncertainty--aware Knowledge Integration (MUKI) framework to recover the golden supervision for the student.
  arXiv  Detail & Related papers  (2022-10-11T07:59:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.