Related papers: Socratic Mind: Impact of a Novel GenAI-Powered Assessment Tool on Student Learning and Higher-Order Thinking

Socratic Mind: Impact of a Novel GenAI-Powered Assessment Tool on Student Learning and Higher-Order Thinking

URL: http://arxiv.org/abs/2509.16262v2
Date: Thu, 16 Oct 2025 04:07:31 GMT
Title: Socratic Mind: Impact of a Novel GenAI-Powered Assessment Tool on Student Learning and Higher-Order Thinking
Authors: Jeonghyun Lee, Jui-Tse Hung, Meryem Yilmaz Soylu, Diana Popescu, Christopher Zhang Cui, Gayane Grigoryan, David A Joyner, Stephen W Harmon,
Abstract summary: This study examines the impact of Socratic Mind, a Generative Artificial Intelligence (GenAI) powered formative assessment tool on learning outcomes.<n>Students who engaged with the GenAI tool experienced significant gains in their quiz scores compared to those who did not.<n>Our findings highlight the promise of AI-mediated dialogue in fostering deeper engagement and higher-order cognitive skills.
Score: 2.192176712066146
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Abstract: This study examines the impact of Socratic Mind, a Generative Artificial Intelligence (GenAI) powered formative assessment tool that employs Socratic questioning to support student learning in a large, fully online undergraduate-level computing course. Employing a quasi-experimental, mixed-methods design, we investigated participants' engagement patterns, the influence of user experience on engagement, and impacts on both perceived and actual learning outcomes. Data were collected from the system logs, surveys on user experience and perceived engagement and learning gains, student reflections, and course performance data. Results indicated that participants consistently reported high levels of affective, behavioral, and cognitive engagement, and these were strongly linked to positive user experiences and perceived learning outcomes. Quantitative analysis further revealed that students who engaged with the GenAI tool experienced significant gains in their quiz scores compared to those who did not, particularly benefiting students with lower baseline achievement. Additionally, thematic analysis of qualitative feedback revealed substantial perceived improvements in higher-order thinking skills, including problem solving, critical thinking, and self-reflection. Our findings highlight the promise of AI-mediated dialogue in fostering deeper engagement and higher-order cognitive skills. As higher education institutions expand GenAI integration in curriculum, this dialogic, GenAI powered assessment tool can offer a scalable strategy to promote students' meaningful learning outcomes.

Related papers

AI Literacy, Safety Awareness, and STEM Career Aspirations of Australian Secondary Students: Evaluating the Impact of Workshop Interventions [38.350232667249095]
Deepfakes and other forms of synthetic media pose growing safety risks for adolescents.<n>This study evaluates the impact of Day of AI Australia's workshop-based intervention on Australian secondary students.
arXiv Detail & Related papers (2026-01-30T02:55:53Z)
From Co-Design to Metacognitive Laziness: Evaluating Generative AI in Vocational Education [0.0]
This study examines the development and deployment of a Generative AI proof-of-concept (POC) designed to support lecturers in a vocational education setting in Singapore.<n>The POC achieved its primary operational goals: lecturers reported streamlined, reduced cognitive load, and observed improved student confidence in navigating course content.<n>Despite enhanced teaching processes, performance data revealed no significant improvement in overall student assessment outcomes.<n>The study raises consequential design questions regarding how AI tools can be engineered to minimise dependency, metacognitive development, and calibrate support across varying ability levels.
arXiv Detail & Related papers (2025-12-13T12:26:25Z)
AI Literacy as a Key Driver of User Experience in AI-Powered Assessment: Insights from Socratic Mind [2.0272430076690027]
This study examines how students' AI literacy and prior exposure to AI technologies shape their perceptions of Socratic Mind.<n>Data from 309 undergraduates in Computer Science and Business courses were collected.
arXiv Detail & Related papers (2025-07-29T10:11:24Z)
ChatGPT produces more "lazy" thinkers: Evidence of cognitive engagement decline [0.0]
This study investigates the impact of generative artificial intelligence (AI) tools on the cognitive engagement of students during academic writing tasks.<n>The results revealed significantly lower cognitive engagement scores in the ChatGPT group compared to the control group.<n>These findings suggest that AI assistance may lead to cognitive offloading.
arXiv Detail & Related papers (2025-06-30T18:41:50Z)
Evaluating the AI-Lab Intervention: Impact on Student Perception and Use of Generative AI in Early Undergraduate Computer Science Courses [0.0]
Generative AI (GenAI) is rapidly entering computer science education.<n>Concerns about overreliance coexist with a gap in research on structured scaffolding to guide tool use in formal courses.<n>This study examines the impact of a dedicated "AI-Lab" intervention on undergraduate students.
arXiv Detail & Related papers (2025-04-30T18:12:42Z)
Level Up Peer Review in Education: Investigating genAI-driven Gamification system and its influence on Peer Feedback Effectiveness [0.8087870525861938]
This paper introduces Socratique, a gamified peer-assessment platform integrated with Generative AI (GenAI) assistance.<n>By incorporating game elements, Socratique aims to motivate students to provide more feedback.<n>Students in the treatment group provided significantly more voluntary feedback, with higher scores on clarity, relevance, and specificity.
arXiv Detail & Related papers (2025-04-03T18:30:25Z)
Analyzing the Impact of AI Tools on Student Study Habits and Academic Performance [0.0]
The research focuses on how AI tools can support personalized learning, adaptive test adjustments, and provide real-time classroom analysis.<n>Student feedback revealed strong support for these features, and the study found a significant reduction in study hours alongside an increase in GPA.<n>Despite these benefits, challenges such as over-reliance on AI and difficulties in integrating AI with traditional teaching methods were also identified.
arXiv Detail & Related papers (2024-12-03T04:51:57Z)
Evaluating and Optimizing Educational Content with Large Language Model Judgments [52.33701672559594]
We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
arXiv Detail & Related papers (2024-03-05T09:09:15Z)
Toward enriched Cognitive Learning with XAI [44.99833362998488]
We introduce an intelligent system (CL-XAI) for Cognitive Learning which is supported by artificial intelligence (AI) tools. The use of CL-XAI is illustrated with a game-inspired virtual use case where learners tackle problems to enhance problem-solving skills.
arXiv Detail & Related papers (2023-12-19T16:13:47Z)
Towards Goal-oriented Intelligent Tutoring Systems in Online Education [65.17234980710386]
We propose a new task, named Goal-oriented Intelligent Tutoring Systems (GITS)<n>GITS aims to enable the student's mastery of a designated concept by strategically planning a customized sequence of exercises and assessment.<n>We propose a novel graph-based reinforcement learning framework, named Planning-Assessment-Interaction (PAI)
arXiv Detail & Related papers (2023-12-03T12:37:16Z)
Comparative Study of Learning Outcomes for Online Learning Platforms [47.5164159412965]
Personalization and active learning are key aspects to successful learning. We run a comparative head-to-head study of learning outcomes for two popular online learning platforms.
arXiv Detail & Related papers (2021-04-15T20:40:24Z)
Personalized Education in the AI Era: What to Expect Next? [76.37000521334585]
The objective of personalized learning is to design an effective knowledge acquisition track that matches the learner's strengths and bypasses her weaknesses to meet her desired goal. In recent years, the boost of artificial intelligence (AI) and machine learning (ML) has unfolded novel perspectives to enhance personalized education.
arXiv Detail & Related papers (2021-01-19T12:23:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.