A Multi-Agent Approach to Validate and Refine LLM-Generated Personalized Math Problems

• URL: http://arxiv.org/abs/2604.05160v1
• Date: Mon, 06 Apr 2026 20:47:01 GMT
• Title: A Multi-Agent Approach to Validate and Refine LLM-Generated Personalized Math Problems
• Abstract summary: We propose a framework that formalizes personalization as an iterative generate--validate--revise process.<n>We use four specialized validator agents targeting the criteria of solvability, realism, readability, and authenticity.<n>We evaluate our framework on 600 problems drawn from a popular online mathematics homework platform, ASSISTments.
• Score: 5.687145473906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Students benefit from math problems contextualized to their interests. Large language models (LLMs) offer promise for efficient personalization at scale. However, LLM-generated personalized problems may often have problems such as unrealistic quantities and contexts, poor readability, limited authenticity with respect to students' experiences, and occasional mathematical inconsistencies. To alleviate these problems, we propose a multi-agent framework that formalizes personalization as an iterative generate--validate--revise process; we use four specialized validator agents targeting the criteria of solvability, realism, readability, and authenticity, respectively. We evaluate our framework on 600 problems drawn from a popular online mathematics homework platform, ASSISTments, personalizing each problem to a fixed set of 20 student interest topics. We compare three refinement strategies that differ in how validation feedback is coordinated into revisions. Results show that authenticity and realism are the most frequent failure modes in initial LLM-personalized problems, but that a single refinement iteration substantially reduces these failures. We further find that different refinement strategies have different strengths on different criteria. We also assess validator reliability via human evaluation. Results show that reliability is highest on realism and lowest on authenticity, highlighting the need for better evaluation protocols that consider teachers' and students' personal characteristics.

Related papers

• Simulating Students' Java Programming Errors with Large Language Models [7.543318372822572]
  This paper explores whether large language models (LLMs) can serve as scalable proxies for students by simulating realistic logical errors in code submissions.<n>Using the CodeWorkout dataset of 74,000+ unique student Java submissions across 37 problems, we evaluate five LLMs under three mainstream prompting strategies.<n>Our quantitative findings reveal that while all models generate diverse errors, their alignment to human submissions diverges: Claude Sonnet 4 achieves the most balanced performance.
  arXiv  Detail & Related papers  (2026-06-12T04:51:49Z)
• Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs [2.727845763576068]
  Vision-capable large language models (LLMs) offer new opportunities.<n>We present an empirical evaluation of an LLM-based grader for handwritten mathematical work using instructor-defined rubrics.
  arXiv  Detail & Related papers  (2026-05-18T19:06:15Z)
• Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math [55.83696908107408]
  We introduce ScratchMath, a novel benchmark for explaining and classifying errors in authentic handwritten mathematics scratchwork.<n>Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students.<n>We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts.
  arXiv  Detail & Related papers  (2026-03-26T02:57:20Z)
• Reward Modeling for Scientific Writing Evaluation [50.33952894976367]
  It is critical to develop models that can be reliably deployed for evaluating diverse open-ended scientific writing tasks.<n>We propose cost-efficient, open-source reward models tailored for scientific writing evaluation.
  arXiv  Detail & Related papers  (2026-01-16T15:32:58Z)
• SMART: Self-Generating and Self-Validating Multi-Dimensional Assessment for LLMs' Mathematical Problem Solving [24.689620248781214]
  Large Language Models (LLMs) have achieved remarkable results on a variety of mathematical benchmarks.<n>Common evaluation methods, which focus on the either the final answer or the reasoning process, fail to assess the entire problem-solving procedure.<n>Our findings reveal genuine weaknesses in current LLMs and motivate a new metric, the All-Pass Score, to better capture true problem-solving capabilities.
  arXiv  Detail & Related papers  (2025-05-22T13:18:24Z)
• Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications [0.0]
  Large Language Models (LLMs) have demonstrated impressive performance across diverse domains, yet they still encounter challenges such as insufficient domain-specific knowledge, biases, and hallucinations.<n>Traditional evaluation methods, which rely on word overlap or text embeddings, are inadequate for capturing the nuanced semantic information necessary to evaluate dynamic, open-ended text generation.<n>We propose a novel dynamic multi-agent system that automatically designs personalized LLM judges for various natural language generation applications.
  arXiv  Detail & Related papers  (2025-04-01T09:36:56Z)
• Self-Evolving Critique Abilities in Large Language Models [59.861013614500024]
  This paper explores enhancing critique abilities of Large Language Models (LLMs)<n>We introduce SCRIT, a framework that trains LLMs with self-generated data to evolve their critique abilities.<n>Our analysis reveals that SCRIT's performance scales positively with data and model size.
  arXiv  Detail & Related papers  (2025-01-10T05:51:52Z)
• Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors [78.53699244846285]
  Large language models (LLMs) present an opportunity to scale high-quality personalized education to all. LLMs struggle to precisely detect student's errors and tailor their feedback to these errors. Inspired by real-world teaching practice where teachers identify student errors and customize their response based on them, we focus on verifying student solutions.
  arXiv  Detail & Related papers  (2024-07-12T10:11:40Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models.<n>It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Evaluating Mathematical Reasoning Beyond Accuracy [50.09931172314218]
  We introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps.<n>We show that ReasonEval consistently outperforms baseline methods in the meta-evaluation datasets.<n>We observe that ReasonEval can play a significant role in data selection.
  arXiv  Detail & Related papers  (2024-04-08T17:18:04Z)
• MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation [60.65820977963331]
  We introduce a novel evaluation paradigm for Large Language Models (LLMs) This paradigm shifts the emphasis from result-oriented assessments, which often neglect the reasoning process, to a more comprehensive evaluation. By applying this paradigm in the GSM8K dataset, we have developed the MR-GSM8K benchmark.
  arXiv  Detail & Related papers  (2023-12-28T15:49:43Z)
• Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks [65.69651759036535]
  We analyze whether large language models (LLMs) can serve as reliable alternatives to humans.<n>This paper explores both conventional tasks (e.g., story generation) and alignment tasks (e.g., math reasoning)<n>We find that LLM evaluators can generate unnecessary criteria or omit crucial criteria, resulting in a slight deviation from the experts.
  arXiv  Detail & Related papers  (2023-10-30T17:04:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.