The Conversational Exam: A Scalable Assessment Design for the AI Era

• URL: http://arxiv.org/abs/2601.10691v1
• Date: Thu, 15 Jan 2026 18:50:48 GMT
• Title: The Conversational Exam: A Scalable Assessment Design for the AI Era
• Abstract summary: This paper presents a scalable oral examination format that restores assessment validity by having students code live while explaining their reasoning.<n>We examined 58 students in small groups across just two days, demonstrating that oral exams can scale to typical class sizes.
• Score: 1.4323566945483497
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional assessment methods collapse when students use generative AI to complete work without genuine engagement, creating an illusion of competence where they believe they're learning but aren't. This paper presents the conversational exam -- a scalable oral examination format that restores assessment validity by having students code live while explaining their reasoning. Drawing on human-computer interaction principles, we examined 58 students in small groups across just two days, demonstrating that oral exams can scale to typical class sizes. The format combines authentic practice (students work with documentation and supervised AI access) with inherent validity (real-time performance cannot be faked). We provide detailed implementation guidance to help instructors adapt this approach, offering a practical path forward when many educators feel paralyzed between banning AI entirely or accepting that valid assessment is impossible.

Related papers

• Socrates went Nuclear: Comparing Interaction Strategies for AI systems in a Learning Context using Brain Sensing [66.03354630509942]
  This paper reports on a study where we compare three designs for user-AI interaction in a learning context.<n>Fifty study participants were tasked with learning about nuclear safety protocols.
  arXiv  Detail & Related papers  (2026-09-01T02:24:35Z)
• LearnAI: Just-in-Time AI Co-Creation Across Disciplines at a University [2.980329026942995]
  This report presents the LearnAI Framework, a two-layer model for just-in-time AI co-creation piloted at a comprehensive teaching university.<n>The Wide-Exposure Layer embeds short presentations in existing courses to build AI awareness at scale, reaching students and faculty across 18 courses in five disciplines.<n>The Customized Co-Creation Layer provides opt-in, one-on-one sessions where clients work with trained undergraduate tutors through a 5-Stage Pedagogical Script.<n>Interviews with five clients and two tutors suggest a recurring change in how clients described AI use, shifting from treating AI as a passive answer machine to engaging
  arXiv  Detail & Related papers  (2026-08-19T17:44:54Z)
• Student Evaluation of Repeated AI Feedback Across a Semester of Writing [0.0]
  Students obtained and assessed feedback from a self-selected AI tool using a uniform prompt.<n>Students found feedback helpful and actionable more often than not.<n>The study contributes descriptive classroom evidence on integration of AI feedback.
  arXiv  Detail & Related papers  (2026-07-17T16:53:00Z)
• Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AI [47.027290803102666]
  We propose CoRe-3 (Co-Reasoning), a competency model factoring productive AI use into three assessable skills.<n>We ground the skills in theory, state five testable propositions, and instantiate them in CoReasoningLab.<n>Human-rater agreement and outcomes are next; we release the instrument, data, and protocol.
  arXiv  Detail & Related papers  (2026-06-04T10:25:31Z)
• Characterizing initial human-AI proof formalization workflows [66.06866890889131]
  Advances in AI systems' ability to generate code and engage in increasingly high-level mathematical reasoning promise to transform people's ability to formalize and thereby verify proofs.<n>We conduct a mixed-methods analysis into the initial impact of AI on people's formalization.<n>A qualitative survey shows that people's preferences are diverse, but with a general desire for AI assistance in formalization.
  arXiv  Detail & Related papers  (2026-06-02T22:58:19Z)
• Reimagining Assessment in the Age of Generative AI: Lessons from Open-Book Exams with ChatGPT [1.2183405753834562]
  Generative AI systems such as ChatGPT challenge traditional assumptions about academic assessment by enabling students to generate explanations, code, and solutions in real time.<n>Rather than attempting to restrict AI use, this study investigates how students actually interact with such systems during formal evaluation.
  arXiv  Detail & Related papers  (2026-05-12T16:33:33Z)
• UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models [59.693733170193944]
  Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings.<n>Recent reinforcement learning approaches address this limitation but face two critical challenges.<n>We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges.
  arXiv  Detail & Related papers  (2025-11-12T01:27:02Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• Existing Large Language Model Unlearning Evaluations Are Inconclusive [105.55899615056573]
  We show that some evaluations introduce substantial new information into the model, potentially masking true unlearning performance.<n>We demonstrate that evaluation outcomes vary significantly across tasks, undermining the generalizability of current evaluation routines.<n>We propose two principles for future unlearning evaluations: minimal information injection and downstream task awareness.
  arXiv  Detail & Related papers  (2025-05-31T19:43:00Z)
• From Coders to Critics: Empowering Students through Peer Assessment in the Age of AI Copilots [3.3094795918443634]
  This paper presents an empirical study of a rubric based, anonymized peer review process implemented in a large programming course.<n>Students evaluated each other's final projects (2D game) and their assessments were compared to instructor grades using correlation, mean absolute error, and root mean square error (RMSE)<n>Results show that peer review can approximate instructor evaluation with moderate accuracy and foster student engagement, evaluative thinking, and interest in providing good feedback to their peers.
  arXiv  Detail & Related papers  (2025-05-28T08:17:05Z)
• Supporting Students' Reading and Cognition with AI [12.029238454394445]
  We analyzed text from 124 sessions with AI tools to understand users' reading processes and cognitive engagement.<n>We propose design implications for future AI reading-support systems, including structured scaffolds for lower-level cognitive tasks.<n>We advocate for adaptive, human-in-the-loop features that allow students and instructors to tailor their reading experiences with AI.
  arXiv  Detail & Related papers  (2025-04-07T17:51:27Z)
• Leveraging Large Language Models for Actionable Course Evaluation Student Feedback to Lecturers [6.161370712594005]
  We have 742 student responses ranging over 75 courses in a Computer Science department. For each course, we synthesise a summary of the course evaluations and actionable items for the instructor. Our work highlights the possibility of using generative AI to produce factual, actionable, and appropriate feedback for teachers in the classroom setting.
  arXiv  Detail & Related papers  (2024-07-01T13:29:55Z)
• Student Mastery or AI Deception? Analyzing ChatGPT's Assessment Proficiency and Evaluating Detection Strategies [1.633179643849375]
  Generative AI systems such as ChatGPT have a disruptive effect on learning and assessment. This work investigates the performance of ChatGPT by evaluating it across three courses.
  arXiv  Detail & Related papers  (2023-11-27T20:10:13Z)
• Unsupervised Sentiment Analysis of Plastic Surgery Social Media Posts [91.3755431537592]
  The massive collection of user posts across social media platforms is primarily untapped for artificial intelligence (AI) use cases. Natural language processing (NLP) is a subfield of AI that leverages bodies of documents, known as corpora, to train computers in human-like language understanding. This study demonstrates that the applied results of unsupervised analysis allow a computer to predict either negative, positive, or neutral user sentiment towards plastic surgery.
  arXiv  Detail & Related papers  (2023-07-05T20:16:20Z)
• Perception, performance, and detectability of conversational artificial intelligence across 32 university courses [15.642614735026106]
  We compare the performance of ChatGPT against students on 32 university-level courses. We find that ChatGPT's performance is comparable, if not superior, to that of students in many courses. We find an emerging consensus among students to use the tool, and among educators to treat this as plagiarism.
  arXiv  Detail & Related papers  (2023-05-07T10:37:51Z)
• TEMPERA: Test-Time Prompting via Reinforcement Learning [57.48657629588436]
  We propose Test-time Prompt Editing using Reinforcement learning (TEMPERA) In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.
  arXiv  Detail & Related papers  (2022-11-21T22:38:20Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.