Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT

• URL: http://arxiv.org/abs/2605.12363v1
• Date: Tue, 12 May 2026 16:33:33 GMT
• Title: Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT
• Abstract summary: Generative AI systems such as ChatGPT challenge traditional assumptions about academic assessment by enabling students to generate explanations, code, and solutions in real time.<n>Rather than attempting to restrict AI use, this study investigates how students actually interact with such systems during formal evaluation.
• Score: 1.2183405753834562
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Generative AI systems such as ChatGPT challenge traditional assumptions about academic assessment by enabling students to generate explanations, code, and solutions in real time. Rather than attempting to restrict AI use, this study investigates how students actually interact with such systems during formal evaluation. Engineering students were permitted to use ChatGPT during take-home open-book exams and were required to submit interaction transcripts alongside exam solutions. This provided direct observational evidence of reasoning processes rather than relying on self-reported behavior. Qualitative analysis revealed three progressive patterns of use: answer retrieval, guided collaboration, and critical verification. While some students initially copied questions verbatim and received generic responses, many refined prompts iteratively and tested outputs. Some of the strongest evidence of reasoning appeared when students evaluated incorrect or incomplete AI responses, revealing evaluative reasoning through debugging, comparison, and justification. The presence of generative AI shifted the cognitive task of assessment from producing solutions to assessing solution validity. The findings suggest that, in AI-mediated assessment environments, correctness of final answers alone may no longer provide sufficient evidence of comprehension. Instead, competencies such as prompt formulation, verification, and judgment become visible indicators of learning. Transparent integration of AI appeared to reduce focus on rule avoidance and promote self-regulation. Assessments should evolve to evaluate reasoning about solutions rather than independent solution production. Generative AI therefore does not invalidate assessment but has the potential to expose deeper forms of understanding aligned with professional practice.

Related papers

• Who Should Grade My Work? Student Perspectives on Transparent AI-Assisted Writing Assessment in Higher Education [0.0]
  The integration of GenAI tools into higher education assessment raises important questions about how students understand, interpret, and respond to AI-mediated evaluation.<n>This study reports findings from a qualitative pedagogical inquiry conducted in an undergraduate technical communication course for computing students at a Saudi public university.<n>Students were explicitly informed that ChatGPT had generated the score and feedback and were invited to reflect on the evaluation in writing.
  arXiv  Detail & Related papers  (2026-09-04T16:49:33Z)
• Socrates went Nuclear: Comparing Interaction Strategies for AI systems in a Learning Context using Brain Sensing [66.03354630509942]
  This paper reports on a study where we compare three designs for user-AI interaction in a learning context.<n>Fifty study participants were tasked with learning about nuclear safety protocols.
  arXiv  Detail & Related papers  (2026-09-01T02:24:35Z)
• Evaluation in the Age of AI: Output as Evidence of Learning [0.0]
  The rapid adoption of artificial intelligence (AI) has disrupted how learning is demonstrated and evaluated in higher education.<n>This paper examines the ethical challenges of educational evaluation in the age of AI from a university-level perspective.
  arXiv  Detail & Related papers  (2026-08-23T23:40:30Z)
• A Classroom Study of LLM-Generated Feedback Intervention in Introductory Programming [1.0241350366020698]
  We present a large-scale classroom study in which AI-generated feedback was deployed in an introductory Python programming course.<n>Students received one of three feedback conditions on incorrect submissions: natural language hints, AI-generated failing test cases, or no AI feedback.<n>We find that natural language feedback is significantly associated with higher completion rates and faster convergence to correct solutions.
  arXiv  Detail & Related papers  (2026-06-07T19:55:51Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• Trust and Reliance on AI in Education: AI Literacy and Need for Cognition as Moderators [2.0391237204597363]
  We investigate how students' trust relates to their appropriate reliance on an AI assistant during programming problem-solving tasks.<n>We operationalize reliance behaviorally as the extent to which students' responses reflected appropriate use of the AI assistant's suggestions.
  arXiv  Detail & Related papers  (2026-04-01T16:38:47Z)
• Strategies for Creating Uncertainty in the AI Era to Trigger Students Critical Thinking: Pedagogical Design, Assessment Rubric, and Exam System [0.0]
  Generative exam AI challenges traditional assessments by allowing students to produce correct answers without demonstrating understanding or reasoning.<n>Rather than AI, this work argues that one way to integrate AI into education is by creating uncertain situations with help of AI models.<n>We introduce MindsaicAIExam, an exam system that integrates AI tools and requires students to provide initial answers, critically evaluate AI outputs, and refine their reasoning.
  arXiv  Detail & Related papers  (2026-01-17T10:29:43Z)
• The Conversational Exam: A Scalable Assessment Design for the AI Era [1.4323566945483497]
  This paper presents a scalable oral examination format that restores assessment validity by having students code live while explaining their reasoning.<n>We examined 58 students in small groups across just two days, demonstrating that oral exams can scale to typical class sizes.
  arXiv  Detail & Related papers  (2026-01-15T18:50:48Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Existing Large Language Model Unlearning Evaluations Are Inconclusive [105.55899615056573]
  We show that some evaluations introduce substantial new information into the model, potentially masking true unlearning performance.<n>We demonstrate that evaluation outcomes vary significantly across tasks, undermining the generalizability of current evaluation routines.<n>We propose two principles for future unlearning evaluations: minimal information injection and downstream task awareness.
  arXiv  Detail & Related papers  (2025-05-31T19:43:00Z)
• Beyond Detection: Designing AI-Resilient Assessments with Automated Feedback Tool to Foster Critical Thinking [0.0]
  This research proposes a proactive, AI-resilient solution based on assessment design rather than detection.<n>It introduces a web-based Python tool that integrates Bloom's taxonomy with advanced natural language processing techniques.<n>It helps educators determine whether a task targets lower-order thinking such as recall and summarization or higher-order skills such as analysis, evaluation, and creation.
  arXiv  Detail & Related papers  (2025-03-30T23:13:00Z)
• AERA Chat: An Interactive Platform for Automated Explainable Student Answer Assessment [15.969280805269976]
  AERA Chat is an interactive visualization platform designed for automated explainable student answer assessment.<n>AERA Chat leverages multiple language models (LLMs) to concurrently score student answers and generate explanatory rationales.<n>We demonstrate the effectiveness of our platform through evaluations of multiple rationale-generation methods on several datasets.
  arXiv  Detail & Related papers  (2024-10-12T11:57:53Z)
• Evaluating the Utility of Model Explanations for Model Development [54.23538543168767]
  We evaluate whether explanations can improve human decision-making in practical scenarios of machine learning model development. To our surprise, we did not find evidence of significant improvement on tasks when users were provided with any of the saliency maps. These findings suggest caution regarding the usefulness and potential for misunderstanding in saliency-based explanations.
  arXiv  Detail & Related papers  (2023-12-10T23:13:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.