Baseline Performance of AI Tools in Classifying Cognitive Demand of Mathematical Tasks

• URL: http://arxiv.org/abs/2603.03512v1
• Date: Tue, 03 Mar 2026 20:39:55 GMT
• Title: Baseline Performance of AI Tools in Classifying Cognitive Demand of Mathematical Tasks
• Abstract summary: We tested eleven AI tools on their ability to categorize mathematics tasks across four levels of cognitive demand.<n>On average, AI tools accurately classified cognitive demand in only 63% of cases.<n>All tools struggled with tasks at the extremes of cognitive demand.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Teachers face increasing demands on their time, particularly in adapting mathematics curricula to meet individual student needs while maintaining cognitive rigor. This study evaluates whether AI tools can accurately classify the cognitive demand of mathematical tasks, which is important for creating or adapting tasks that support student learning. We tested eleven AI tools: six general-purpose (ChatGPT, Claude, DeepSeek, Gemini, Grok, Perplexity) and five education-specific (Brisk, Coteach AI, Khanmigo, Magic School, School.AI), on their ability to categorize mathematics tasks across four levels of cognitive demand using a research-based framework. The goal was to approximate the performance teachers will achieve with straightforward prompts. On average, AI tools accurately classified cognitive demand in only 63% of cases. Education-specific tools were not more accurate than general-purpose tools, and no tool exceeded 83% accuracy. All tools struggled with tasks at the extremes of cognitive demand (Memorization and Doing Mathematics), exhibiting a systematic bias toward middle-category levels (Procedures with/without Connections). The tools often gave plausible-sounding explanations likely to be persuasive to novice teachers. Error analysis of AI tools' misclassification of the broad level of cognitive demand (high vs. low) revealed that tools consistently overweighted surface textual features over underlying cognitive processes. Further, AI tools showed weaknesses in reasoning about factors that make tasks higher vs. lower cognitive demand. Errors stemmed not from ignoring relevant dimensions, but from incorrectly reasoning about multiple task aspects. These findings carry implications for AI integration into teacher planning workflows and highlight the need for improved prompt engineering and tool development for educational applications.

Related papers

• Characterizing initial human-AI proof formalization workflows [66.06866890889131]
  Advances in AI systems' ability to generate code and engage in increasingly high-level mathematical reasoning promise to transform people's ability to formalize and thereby verify proofs.<n>We conduct a mixed-methods analysis into the initial impact of AI on people's formalization.<n>A qualitative survey shows that people's preferences are diverse, but with a general desire for AI assistance in formalization.
  arXiv  Detail & Related papers  (2026-06-02T22:58:19Z)
• Temporal Stability and Few-Shot Prompting in Math Task Assessment [0.0]
  This study focused on different AI tools' ability to use the Task Analysis Guide to classify cognitive demand of mathematics tasks.<n>We tested a general-purpose AI tool (Gemini) and an education-specific AI tool (Coteach)<n>Results revealed that newer model versions alone produced mixed effects: Gemini's accuracy remained stable at 58%, while Coteach's accuracy decreased from 75% to 50%.
  arXiv  Detail & Related papers  (2026-05-28T16:10:24Z)
• Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work [9.434503228899455]
  We introduce a course-based practice that teaches AI through benchmark construction.<n>Students turn disciplinary knowledge into verifiable expert-level questions.<n>Students evaluate AI systems on the resulting tasks.
  arXiv  Detail & Related papers  (2026-05-20T17:09:56Z)
• Intentmaking and Sensemaking: Human Interaction with AI-Guided Mathematical Discovery [72.49593499512092]
  We identify and characterize a distinct workflow we term intentmaking.<n>We frame this as a natural extension to sensemaking.<n>Our documentation of these themes suggests an approach to designing AI tools for scientific discovery.
  arXiv  Detail & Related papers  (2026-05-07T09:30:25Z)
• Can AI Tools Transform Low-Demand Math Tasks? An Evaluation of Task Modification Capabilities [0.0]
  This study investigated whether AI tools could successfully upgrade low-cognitive-demand mathematics tasks.<n>On average, AI tools were only moderately successful: tasks were accurately upgraded only 64% of the time.<n> Specialized tools were only moderately more successful than general-purpose tools.
  arXiv  Detail & Related papers  (2026-04-14T13:57:09Z)
• Perceived Importance of Cognitive Skills Among Computing Students in the Era of AI [2.0391237204597363]
  The availability and increasing integration of generative AI tools have transformed computing education.<n>While AI in education presents opportunities, it also raises new concerns about how these powerful know-it-all AI tools, which are becoming widespread, impact cognitive skill development among students.
  arXiv  Detail & Related papers  (2026-04-12T17:05:26Z)
• AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoning [66.24374176797075]
  We introduce textbfAdaReasoner, a family of multimodal models that learn tool use as a general reasoning skill rather than as tool-specific or explicitly supervised behavior.<n>AdaReasoner is enabled by (i) a scalable data curation pipeline exposing models to long-horizon, multi-step tool interactions; (ii) Tool-GRPO, a reinforcement learning algorithm that prioritizes tool selection and sequencing based on end-task success; and (iii) an adaptive learning mechanism that dynamically regulates tool usage.
  arXiv  Detail & Related papers  (2026-01-26T16:04:43Z)
• What Work is AI Actually Doing? Uncovering the Drivers of Generative AI Adoption [1.4977849232424492]
  This study investigates which intrinsic task characteristics drive users' decisions to delegate work to AI systems.<n>This research provides the first systematic evidence linking real-world generative AI usage to a comprehensive, multi-dimensional framework of intrinsic task characteristics.
  arXiv  Detail & Related papers  (2025-10-26T19:13:37Z)
• AI Agents as Universal Task Solvers [94.49762121230042]
  We show that the optimal speed-up that a universal solver can achieve using past data is tightly related to their algorithmic information.<n>We argue that the key quantity to optimize when scaling reasoning models is time, whose critical role in learning has so far only been indirectly considered.
  arXiv  Detail & Related papers  (2025-10-14T02:17:54Z)
• Comparative Analysis of STEM and non-STEM Teachers' Needs for Integrating AI into Educational Environments [0.6138671548064355]
  This study explores how educational platforms can be improved by incorporating AI and analytics features.<n>We interviewed 8 K-12 teachers and asked their practices and needs while using any block-based programming (BBP) platform in their classes.
  arXiv  Detail & Related papers  (2025-09-18T16:20:18Z)
• Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence [59.07578850674114]
  Sound deductive reasoning is an indisputably desirable aspect of general intelligence.<n>It is well-documented that even the most advanced frontier systems regularly and consistently falter on easily-solvable reasoning tasks.<n>We argue that their unsound behavior is a consequence of the statistical learning approach powering their development.
  arXiv  Detail & Related papers  (2025-06-30T14:37:50Z)
• Beyond Detection: Designing AI-Resilient Assessments with Automated Feedback Tool to Foster Critical Thinking [0.0]
  This research proposes a proactive, AI-resilient solution based on assessment design rather than detection.<n>It introduces a web-based Python tool that integrates Bloom's taxonomy with advanced natural language processing techniques.<n>It helps educators determine whether a task targets lower-order thinking such as recall and summarization or higher-order skills such as analysis, evaluation, and creation.
  arXiv  Detail & Related papers  (2025-03-30T23:13:00Z)
• General Scales Unlock AI Evaluation with Explanatory and Predictive Power [57.7995945974989]
  benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems.<n>We introduce general scales for AI evaluation that can explain what common AI benchmarks really measure.<n>Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate.
  arXiv  Detail & Related papers  (2025-03-09T01:13:56Z)
• "From Unseen Needs to Classroom Solutions": Exploring AI Literacy Challenges & Opportunities with Project-based Learning Toolkit in K-12 Education [0.3994567502796064]
  There is a growing need to equip K-12 students with AI literacy skills that extend beyond computer science.<n>This paper explores the integration of a Project-Based Learning (PBL) AI toolkit into diverse subject areas, aimed at helping educators teach AI concepts more effectively.
  arXiv  Detail & Related papers  (2024-12-23T03:31:02Z)
• Formal Mathematical Reasoning: A New Frontier in AI [60.26950681543385]
  We advocate for formal mathematical reasoning and argue that it is indispensable for advancing AI4Math to the next level.<n>We summarize existing progress, discuss open challenges, and envision critical milestones to measure future success.
  arXiv  Detail & Related papers  (2024-12-20T17:19:24Z)
• A Benchmark for Math Misconceptions: Bridging Gaps in Middle School Algebra with AI-Supported Instruction [0.0]
  This study introduces an evaluation benchmark for middle school algebra to be used in artificial intelligence based educational platforms.<n>The data set comprises 55 misconceptions about algebra, common errors, and 220 diagnostic examples.<n>Four out of five educators expressed interest in using the data set with AI to diagnose student misconceptions or train teachers.
  arXiv  Detail & Related papers  (2024-12-04T23:10:29Z)
• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains [54.117238759317004]
  Massive Multitask Agent Understanding (MMAU) benchmark features comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents.
  arXiv  Detail & Related papers  (2024-07-18T00:58:41Z)
• Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning [43.83422798569986]
  Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and reliable form of assessment. To date, the task of crafting high-quality distractors has largely remained a labor-intensive process for teachers and learning content designers. We propose a simple, in-context learning-based solution for automated distractor and corresponding feedback message generation.
  arXiv  Detail & Related papers  (2023-08-07T01:03:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.