Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course

• URL: http://arxiv.org/abs/2601.04225v1
• Date: Sun, 28 Dec 2025 22:51:25 GMT
• Title: Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course
• Abstract summary: This paper is a student-led field experiment embedded as a midterm project in UNIV 182 (AI4All) at George Mason University.<n>Students designed their own reasoning tasks, ran them on widely used consumer chatbots representative of current capabilities, and evaluated both (i) answer correctness and (ii) the validity of the reasoning.<n>OpenAI GPT-5 and Claude 4.5 achieved the highest mean answer accuracy (86.2% and 83.8%), followed by Grok 4 (82.5%) and Perplexity (73.1%)
• Score: 0.7515394929245942
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Claims about whether large language model (LLM) chatbots "reason" are typically debated using curated benchmarks and laboratory-style evaluation protocols. This paper offers a complementary perspective: a student-led field experiment embedded as a midterm project in UNIV 182 (AI4All) at George Mason University, a Mason Core course designed for undergraduates across disciplines with no expected prior STEM exposure. Student teams designed their own reasoning tasks, ran them on widely used consumer chatbots representative of current capabilities, and evaluated both (i) answer correctness and (ii) the validity of the chatbot's stated reasoning (for example, cases where an answer is correct but the explanation is not, or vice versa). Across eight teams that reported standardized scores, students contributed 80 original reasoning prompts spanning six categories: pattern completion, transformation rules, spatial/visual reasoning, quantitative reasoning, relational/logic reasoning, and analogical reasoning. These prompts yielded 320 model responses plus follow-up explanations. Aggregating team-level results, OpenAI GPT-5 and Claude 4.5 achieved the highest mean answer accuracy (86.2% and 83.8%), followed by Grok 4 (82.5%) and Perplexity (73.1%); explanation validity showed a similar ordering (81.2%, 80.0%, 77.5%, 66.2%). Qualitatively, teams converged on a consistent error signature: strong performance on short, structured math and pattern items but reduced reliability on spatial/visual reasoning and multi-step transformations, with frequent "sound right but reason wrong" explanations. The assignment's primary contribution is pedagogical: it operationalizes AI literacy as experimental practice (prompt design, measurement, rater disagreement, and interpretability/grounding) while producing a reusable, student-generated corpus of reasoning probes grounded in authentic end-user interaction.

Related papers

• Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT [1.2183405753834562]
  Generative AI systems such as ChatGPT challenge traditional assumptions about academic assessment by enabling students to generate explanations, code, and solutions in real time.<n>Rather than attempting to restrict AI use, this study investigates how students actually interact with such systems during formal evaluation.
  arXiv  Detail & Related papers  (2026-05-12T16:33:33Z)
• ArguAgent: AI-Supported Real-Time Grouping for Productive Argumentation in STEM Classrooms [5.090665056048702]
  ArguAgent is an AI-powered system that creates groups based on students' stances and argumentation skills.<n>It constrains argumentation quality differences to +/-1 level on a validated learning progression.<n>It achieves 95.4% of groups that meet both design criteria, a 3.2x improvement over random assignment.
  arXiv  Detail & Related papers  (2026-04-25T21:38:31Z)
• Pause or Fabricate? Training Language Models for Grounded Reasoning [50.104657152302956]
  We propose Grounded Reasoning via Interactive Reinforcement Learning (GRIL) for grounded reasoning under incomplete information.<n>GRIL decomposes the reasoning process into two stages: clarify and pause, which identifies whether the available information is sufficient, and grounded reasoning, which performs task solving once the necessary premises are established.<n>Experiments on GSM8K-Insufficient and MetaMATH-Insufficient show that GRIL significantly improves premise detection (up to 45%), leading to a 30% increase in task success while reducing average response length by over 20%.
  arXiv  Detail & Related papers  (2026-04-21T16:45:29Z)
• Catching The Correct Answer Trap: Characterising AI Tutor Blind Spots When Analysing Student Reasoning [2.6352293843915606]
  We study a failure mode we call the correct answer trap (CAT): models under-detect misconceptions when students reach a correct answer via flawed reasoning.<n>Analysing real student responses from the Eedi mathematics platform, we show that 71% of these failures concentrate in just two question types.
  arXiv  Detail & Related papers  (2026-04-20T10:53:48Z)
• Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding [5.619746163828999]
  Higher correctness is assumed to produce better human understanding, but this link has not been tested experimentally with controlled levels.<n>We show that not all differences in functional correctness translate to differences in human understanding, underscoring the need to validate functional metrics against human outcomes.
  arXiv  Detail & Related papers  (2026-03-26T09:55:10Z)
• CRAwDAD: Causal Reasoning Augmentation with Dual-Agent Debate [3.2852123901391077]
  We develop a dual-agent debate framework for causal inference.<n>Agents attempt to persuade each other, challenging each other's logic.<n>We show that strong models can still benefit greatly from debate with weaker agents.
  arXiv  Detail & Related papers  (2025-11-28T03:19:35Z)
• Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors [58.65143578052761]
  This paper presents a new method, MISTAKE, that constructs high-quality synthetic examples of reasoning errors.<n>We evaluate MISTAKE on three educational tasks and find that it results in (1) higher accuracy when simulating incorrect student answers.
  arXiv  Detail & Related papers  (2025-10-13T15:10:38Z)
• Thinking Before You Speak: A Proactive Test-time Scaling Approach [54.8205006555199]
  We implement our idea as a reasoning framework, named emphThinking Before You Speak (TBYS)<n>We design a pipeline for automatically collecting and filtering in-context examples for the generation of emphinsights.<n>Experiments on challenging mathematical datasets verify the effectiveness of TBYS.
  arXiv  Detail & Related papers  (2025-08-26T03:43:32Z)
• Towards Transparent AI Grading: Semantic Entropy as a Signal for Human-AI Disagreement [2.6293270655263385]
  We introduce semantic entropy, a measure of variability across multiple GPT-4-generated explanations for the same student response, as a proxy for human grader disagreement.<n>Our findings position semantic entropy as an interpretable uncertainty signal that supports more transparent and trustworthy AI-assisted grading.
  arXiv  Detail & Related papers  (2025-08-06T06:02:14Z)
• Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers [8.936182649298766]
  Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding, reasoning, and problem-solving.<n>This paper examines the reasoning capabilities of contemporary LLMs, analyzing their strengths, limitations, and potential for improvement.
  arXiv  Detail & Related papers  (2025-05-02T16:16:17Z)
• A Pilot Evaluation of ChatGPT and DALL-E 2 on Decision Making and Spatial Reasoning [8.185725740857594]
  Two recently released generative transformer models, ChatGPT and DALL-E 2, are evaluated. DALL-E 2 is able to generate at least one correct image for each spatial reasoning prompt, but most images generated are incorrect. In evaluating ChatGPT on the axioms developed under the classical Von Neumann-Morgenstern utility theorem, we find that, although it demonstrates some level of rational decision-making, many of its decisions violate at least one of the axioms.
  arXiv  Detail & Related papers  (2023-02-15T05:04:49Z)
• Lila: A Unified Benchmark for Mathematical Reasoning [59.97570380432861]
  LILA is a unified mathematical reasoning benchmark consisting of 23 diverse tasks along four dimensions. We construct our benchmark by extending 20 datasets benchmark by collecting task instructions and solutions in the form of Python programs. We introduce BHASKARA, a general-purpose mathematical reasoning model trained on LILA.
  arXiv  Detail & Related papers  (2022-10-31T17:41:26Z)
• Making Large Language Models Better Reasoners with Step-Aware Verifier [49.16750018427259]
  DIVERSE (Diverse Verifier on Reasoning Step) is a novel approach that further enhances the reasoning capability of language models. We evaluate DIVERSE on the latest language model code-davinci and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks.
  arXiv  Detail & Related papers  (2022-06-06T03:38:36Z)
• The Unreliability of Explanations in Few-Shot In-Context Learning [50.77996380021221]
  We focus on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We show that explanations judged as good by humans--those that are logically consistent with the input--usually indicate more accurate predictions. We present a framework for calibrating model predictions based on the reliability of the explanations.
  arXiv  Detail & Related papers  (2022-05-06T17:57:58Z)
• PRover: Proof Generation for Interpretable Reasoning over Rules [81.40404921232192]
  We propose a transformer-based model that answers binary questions over rule-bases and generates the corresponding proofs. Our model learns to predict nodes and edges corresponding to proof graphs in an efficient constrained training paradigm. We conduct experiments on synthetic, hand-authored, and human-paraphrased rule-bases to show promising results for QA and proof generation.
  arXiv  Detail & Related papers  (2020-10-06T15:47:53Z)
• CS-NLP team at SemEval-2020 Task 4: Evaluation of State-of-the-art NLP Deep Learning Architectures on Commonsense Reasoning Task [3.058685580689605]
  We describe our attempt at SemEval-2020 Task 4 competition: Commonsense Validation and Explanation (ComVE) challenge. Our system uses prepared labeled textual datasets that were manually curated for three different natural language inference subtasks. For the second subtask, which is to select the reason why a statement does not make sense, we stand within the first six teams (93.7%) among 27 participants with very competitive results.
  arXiv  Detail & Related papers  (2020-05-17T13:20:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.