Temporal Stability and Few-Shot Prompting in Math Task Assessment

• URL: http://arxiv.org/abs/2605.30151v1
• Date: Thu, 28 May 2026 16:10:24 GMT
• Title: Temporal Stability and Few-Shot Prompting in Math Task Assessment
• Abstract summary: This study focused on different AI tools' ability to use the Task Analysis Guide to classify cognitive demand of mathematics tasks.<n>We tested a general-purpose AI tool (Gemini) and an education-specific AI tool (Coteach)<n>Results revealed that newer model versions alone produced mixed effects: Gemini's accuracy remained stable at 58%, while Coteach's accuracy decreased from 75% to 50%.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As AI tools become increasingly integrated into educational contexts, questions arise about both their stability over time and their responsiveness to prompt engineering techniques. This longitudinal study focused on different AI tools' ability to use the Task Analysis Guide (TAG; Stein \& Smith, 1998) to classify the cognitive demand of mathematics tasks. In particular, it examined whether this classification ability changed with (1) model version updates over time and (2) few-shot prompting using exemplar tasks. We tested a general-purpose AI tool (Gemini) and an education-specific AI tool (Coteach). The specific tools were selected because of their relatively high performance on relevant published benchmarks and prior task-specific tests. Models were tested at baseline, retested with model version updates, and then tested again using few-shot prompting (two exemplar tasks for each cognitive demand category). Results revealed that newer model versions alone produced mixed effects: Gemini's accuracy remained stable at 58\%, while Coteach's accuracy decreased from 75\% to 50\%. However, few-shot prompting improved both models' performance: Gemini increased to 67\% and Coteach recovered to 75\% accuracy. These findings demonstrate that prompt engineering techniques can have larger and more reliable effects than passive model improvements, and that version updates may not always improve performance on specialized educational tasks. The study has important implications for how educators and researchers should approach AI tool selection, evaluation, and implementation in educational contexts.

Related papers

• Can AI Tools Transform Low-Demand Math Tasks? An Evaluation of Task Modification Capabilities [0.0]
  This study investigated whether AI tools could successfully upgrade low-cognitive-demand mathematics tasks.<n>On average, AI tools were only moderately successful: tasks were accurately upgraded only 64% of the time.<n> Specialized tools were only moderately more successful than general-purpose tools.
  arXiv  Detail & Related papers  (2026-04-14T13:57:09Z)
• Baseline Performance of AI Tools in Classifying Cognitive Demand of Mathematical Tasks [0.0]
  We tested eleven AI tools on their ability to categorize mathematics tasks across four levels of cognitive demand.<n>On average, AI tools accurately classified cognitive demand in only 63% of cases.<n>All tools struggled with tasks at the extremes of cognitive demand.
  arXiv  Detail & Related papers  (2026-03-03T20:39:55Z)
• DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science [31.00353091375463]
  DARE-bench is a benchmark for machine learning modeling and data science instruction.<n>It consists of 6,300 Kaggle-derived tasks and provides both large-scale training data and evaluation sets.<n>Using DARE-bench training tasks for fine-tuning can substantially improve model performance.
  arXiv  Detail & Related papers  (2026-02-27T18:58:57Z)
• AI Agents as Universal Task Solvers [94.49762121230042]
  We show that the optimal speed-up that a universal solver can achieve using past data is tightly related to their algorithmic information.<n>We argue that the key quantity to optimize when scaling reasoning models is time, whose critical role in learning has so far only been indirectly considered.
  arXiv  Detail & Related papers  (2025-10-14T02:17:54Z)
• Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning [51.34470146576741]
  We propose an agent that assembles a task-specific curriculum and applies reinforcement learning to continue training the model for its target task.<n>The test-time curriculum avoids time-consuming human curation of datasets by automatically selecting the most task-relevant data.<n>Our experiments demonstrate that reinforcement learning on a test-time curriculum consistently improves the model on its target tasks.
  arXiv  Detail & Related papers  (2025-10-06T13:07:14Z)
• Benchmark-Driven Selection of AI: Evidence from DeepSeek-R1 [0.0]
  We show that better performance is not always caused by test-time algorithmic improvements or model sizes but also by using impactful benchmarks as curricula for learning.<n>We call this benchmark-driven selection of AI and show its effects on DeepSeek-R1 using our sequential decision-making problem from Humanity's Last Exam.
  arXiv  Detail & Related papers  (2025-08-13T20:15:20Z)
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs [27.07998056454784]
  ReTool enhances long-form reasoning with tool-integrated learning.<n>Model achieves 67% accuracy with 400 training steps.<n>Remarkably, ReTool-32B attains 72.5% accuracy in extended settings.
  arXiv  Detail & Related papers  (2025-04-15T18:10:22Z)
• General Scales Unlock AI Evaluation with Explanatory and Predictive Power [57.7995945974989]
  benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems.<n>We introduce general scales for AI evaluation that can explain what common AI benchmarks really measure.<n>Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate.
  arXiv  Detail & Related papers  (2025-03-09T01:13:56Z)
• Chatbots im Schulunterricht: Wir testen das Fobizz-Tool zur automatischen Bewertung von Hausaufgaben [0.0]
  This study examines the AI-powered grading tool "AI Grading Assistant" by the German company Fobizz.<n>The tool's numerical grades and qualitative feedback are often random and do not improve even when its suggestions are incorporated.<n>The study critiques the broader trend of adopting AI as a quick fix for systemic problems in education.
  arXiv  Detail & Related papers  (2024-12-09T16:50:02Z)
• The Surprising Effectiveness of Test-Time Training for Few-Shot Learning [59.309477460893916]
  Language models (LMs) have shown impressive performance on tasks within their training distribution, but often struggle with structurally novel tasks.<n>We investigate the effectiveness of test-time training (TTT) as a mechanism for improving LMs' reasoning and few-shot learning capabilities.<n>Our findings highlight the limitations of in-context learning for novel tasks and demonstrate the potential of test-time training to enhance language model adaptability.
  arXiv  Detail & Related papers  (2024-11-11T18:59:45Z)
• TOOLVERIFIER: Generalization to New Tools via Self-Verification [69.85190990517184]
  We introduce a self-verification method which distinguishes between close candidates by self-asking contrastive questions during tool selection. Experiments on 4 tasks from the ToolBench benchmark, consisting of 17 unseen tools, demonstrate an average improvement of 22% over few-shot baselines.
  arXiv  Detail & Related papers  (2024-02-21T22:41:38Z)
• Measuring Massive Multitask Language Understanding [79.6985576698597]
  The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. Largest GPT-3 model improves over random chance by almost 20 percentage points on average. Models also have lopsided performance and frequently do not know when they are wrong.
  arXiv  Detail & Related papers  (2020-09-07T17:59:25Z)
• Evaluation Toolkit For Robustness Testing Of Automatic Essay Scoring Systems [64.4896118325552]
  We evaluate the current state-of-the-art AES models using a model adversarial evaluation scheme and associated metrics. We find that AES models are highly overstable. Even heavy modifications(as much as 25%) with content unrelated to the topic of the questions do not decrease the score produced by the models.
  arXiv  Detail & Related papers  (2020-07-14T03:49:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.