Resurrecting Socrates in the Age of AI: A Study Protocol for Evaluating a Socratic Tutor to Support Research Question Development in Higher Education

• URL: http://arxiv.org/abs/2504.06294v1
• Date: Sat, 05 Apr 2025 00:49:20 GMT
• Title: Resurrecting Socrates in the Age of AI: A Study Protocol for Evaluating a Socratic Tutor to Support Research Question Development in Higher Education
• Authors: Ben Degen,
• Abstract summary: This protocol lays out a study grounded in constructivist learning theory to evaluate a novel AI-based Socratic Tutor.<n>The tutor engages students through iterative, reflective questioning, aiming to promote System 2 thinking.<n>This study aims to advance the understanding of how generative AI can be pedagogically aligned to support, not replace, human cognition.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Formulating research questions is a foundational yet challenging academic skill, one that generative AI systems often oversimplify by offering instant answers at the expense of student reflection. This protocol lays out a study grounded in constructivist learning theory to evaluate a novel AI-based Socratic Tutor, designed to foster cognitive engagement and scaffold research question development in higher education. Anchored in dialogic pedagogy, the tutor engages students through iterative, reflective questioning, aiming to promote System 2 thinking and counteract overreliance on AI-generated outputs. In a quasi-experimental design, approximately 80 German pre-service biology teacher students will be randomly assigned to one of two groups: an AI Socratic Tutor condition and an uninstructed chatbot control. Across multiple cycles, students are expected to formulate research questions based on background texts, with quality assessed through double-blind expert review. The study also examines transfer of skills to novel phenomena and captures student perceptions through mixed-methods analysis, including surveys, interviews and reflective journals. This study aims to advance the understanding of how generative AI can be pedagogically aligned to support, not replace, human cognition and offers design principles for human-AI collaboration in education.

Related papers

• Form-Substance Discrimination: Concept, Cognition, and Pedagogy [55.2480439325792]
  This paper examines form-substance discrimination as an essential learning outcome for curriculum development in higher education.<n>We propose practical strategies for fostering this ability through curriculum design, assessment practices, and explicit instruction.
  arXiv  Detail & Related papers  (2025-04-01T04:15:56Z)
• Sakshm AI: Advancing AI-Assisted Coding Education for Engineering Students in India Through Socratic Tutoring and Comprehensive Feedback [1.9841192743072902]
  Existing AI tools for programming education struggle with key challenges, including the lack of Socratic guidance.<n>This study examines 1170 registered participants, analyzing platform logs, engagement trends, and problem-solving behavior to assess Sakshm AI's impact.
  arXiv  Detail & Related papers  (2025-03-16T12:13:29Z)
• The Imitation Game for Educational AI [23.71250100390303]
  We present a novel evaluation framework based on a two-phase Turing-like test.<n>In Phase 1, students provide open-ended responses to questions, revealing natural misconceptions.<n>In Phase 2, both AI and human experts, conditioned on each student's specific mistakes, generate distractors for new related questions.
  arXiv  Detail & Related papers  (2025-02-21T01:14:55Z)
• AI-Enhanced Sensemaking: Exploring the Design of a Generative AI-Based Assistant to Support Genetic Professionals [38.54324092761751]
  Generative AI has the potential to transform knowledge work, but further research is needed to understand how knowledge workers envision using and interacting with generative AI.<n>Our research focused on designing a generative AI assistant to aid genetic professionals in analyzing whole genome sequences (WGS) and other clinical data for rare disease diagnosis.
  arXiv  Detail & Related papers  (2024-12-19T22:54:49Z)
• AI in Education: Rationale, Principles, and Instructional Implications [0.0]
  Generative AI, like ChatGPT, can create human-like content, prompting questions about its educational role.<n>The study emphasizes deliberate strategies to ensure AI complements, not replaces, genuine cognitive effort.
  arXiv  Detail & Related papers  (2024-12-02T14:08:07Z)
• Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA [43.116608441891096]
  Humans outperform AI systems in knowledge-grounded abductive and conceptual reasoning. State-of-the-art LLMs like GPT-4 and LLaMA show superior performance on targeted information retrieval.
  arXiv  Detail & Related papers  (2024-10-09T03:53:26Z)
• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants [176.39275404745098]
  We evaluate whether two AI assistants, GPT-3.5 and GPT-4, can adequately answer assessment questions. GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions. Our results call for revising program-level assessment design in higher education in light of advances in generative AI.
  arXiv  Detail & Related papers  (2024-08-07T12:11:49Z)
• Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with Students [35.05435052195561]
  The increasing use of Artificial Intelligence (AI) by students in learning presents new challenges for assessing their learning outcomes. This paper introduces a co-design study to explore the potential of students' AI usage data as a novel material for assessment.
  arXiv  Detail & Related papers  (2024-01-26T14:49:29Z)
• Collaborative Learning with Artificial Intelligence Speakers (CLAIS): Pre-Service Elementary Science Teachers' Responses to the Prototype [0.5113447003407372]
  The CLAIS system is designed to have 3-4 human learners join an AI speaker to form a small group, where humans and AI are considered as peers participating in the Jigsaw learning process. The CLAIS system was successfully implemented in a Science Education course session with 15 pre-service elementary science teachers.
  arXiv  Detail & Related papers  (2023-12-20T01:19:03Z)
• AI Chatbots as Multi-Role Pedagogical Agents: Transforming Engagement in CS Education [8.898863361318817]
  We develop, implement, and evaluate a novel learning environment enriched with four distinct chatbots. These roles cater to the three innate psychological needs of learners - competence, autonomy, and relatedness. The system embraces an inquiry-based learning paradigm, encouraging students to ask questions, seek solutions, and explore their curiosities.
  arXiv  Detail & Related papers  (2023-08-08T02:13:44Z)
• Human-AI Coevolution [48.74579595505374]
  Coevolution AI is a process in which humans and AI algorithms continuously influence each other. This paper introduces Coevolution AI as the cornerstone for a new field of study at the intersection between AI and complexity science.
  arXiv  Detail & Related papers  (2023-06-23T18:10:54Z)
• Exploring User Perspectives on ChatGPT: Applications, Perceptions, and Implications for AI-Integrated Education [40.38809129759498]
  ChatGPT is most commonly used in the domains of higher education, K-12 education, and practical skills training. On one hand, some users view it as a transformative tool capable of amplifying student self-efficacy and learning motivation. On the other hand, there is a degree of apprehension among concerned users.
  arXiv  Detail & Related papers  (2023-05-22T15:13:14Z)
• Creation and Evaluation of a Pre-tertiary Artificial Intelligence (AI) Curriculum [58.86139968005518]
  The Chinese University of Hong Kong (CUHK)-Jockey Club AI for the Future Project (AI4Future) co-created an AI curriculum for pre-tertiary education. A team of 14 professors with expertise in engineering and education collaborated with 17 principals and teachers from 6 secondary schools to co-create the curriculum. The co-creation process generated a variety of resources which enhanced the teachers knowledge in AI, as well as fostered teachers autonomy in bringing the subject matter into their classrooms.
  arXiv  Detail & Related papers  (2021-01-19T11:26:19Z)
• Thinking Fast and Slow in AI [38.8581204791644]
  This paper proposes a research direction to advance AI which draws inspiration from cognitive theories of human decision making. The premise is that if we gain insights about the causes of some human capabilities that are still lacking in AI, we may obtain similar capabilities in an AI system.
  arXiv  Detail & Related papers  (2020-10-12T20:10:05Z)
• A general framework for scientifically inspired explanations in AI [76.48625630211943]
  We instantiate the concept of structure of scientific explanation as the theoretical underpinning for a general framework in which explanations for AI systems can be implemented. This framework aims to provide the tools to build a "mental-model" of any AI system so that the interaction with the user can provide information on demand and be closer to the nature of human-made explanations.
  arXiv  Detail & Related papers  (2020-03-02T10:32:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.