Related papers: Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems

Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems

URL: http://arxiv.org/abs/2510.22581v1
Date: Sun, 26 Oct 2025 08:44:21 GMT
Title: Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems
Authors: Kaushal Kumar Maurya, Ekaterina Kochmar,
Abstract summary: generative AI (GenAI) models have accelerated the development of large language model (LLM)-powered Intelligent Tutoring Systems (ITSs)<n>However, the progress and impact of these systems remain largely untraceable due to the absence of reliable, universally accepted, and pedagogy-driven evaluation frameworks and benchmarks.<n>Most existing educational dialogue-based ITS evaluations rely on subjective protocols and non-standardized benchmarks, leading to inconsistencies and limited generalizability.<n>This work provides comprehensive state-of-the-art evaluation practices, highlighting associated challenges through real-world case studies from careful and caring AIED research.
Score: 15.954407353419258
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Abstract: The interdisciplinary research domain of Artificial Intelligence in Education (AIED) has a long history of developing Intelligent Tutoring Systems (ITSs) by integrating insights from technological advancements, educational theories, and cognitive psychology. The remarkable success of generative AI (GenAI) models has accelerated the development of large language model (LLM)-powered ITSs, which have potential to imitate human-like, pedagogically rich, and cognitively demanding tutoring. However, the progress and impact of these systems remain largely untraceable due to the absence of reliable, universally accepted, and pedagogy-driven evaluation frameworks and benchmarks. Most existing educational dialogue-based ITS evaluations rely on subjective protocols and non-standardized benchmarks, leading to inconsistencies and limited generalizability. In this work, we take a step back from mainstream ITS development and provide comprehensive state-of-the-art evaluation practices, highlighting associated challenges through real-world case studies from careful and caring AIED research. Finally, building on insights from previous interdisciplinary AIED research, we propose three practical, feasible, and theoretically grounded research directions, rooted in learning science principles and aimed at establishing fair, unified, and scalable evaluation methodologies for ITSs.

Related papers

Rethinking AI Evaluation in Education: The TEACH-AI Framework and Benchmark for Generative AI Assistants [8.591535882390918]
TEACH-AI is a domain-independent, pedagogically grounded, and stakeholder-aligned framework for guiding the design, development, and evaluation of generative AI systems in education.<n>Our work invites the community to reconsider what constructs "effective" AI in education and to design model evaluation approaches that promote co-creation, inclusivity, and long-term human, social, and educational impact.
arXiv Detail & Related papers (2025-11-28T17:42:36Z)
AI-Educational Development Loop (AI-EDL): A Conceptual Framework to Bridge AI Capabilities with Classical Educational Theories [8.500617875591633]
This study introduces the AI-Educational Development Loop (AI-EDL), a theory-driven framework that integrates classical learning theories with human-in-the-loop artificial intelligence (AI)<n>The framework emphasizes transparency, self-regulated learning, and pedagogical oversight.
arXiv Detail & Related papers (2025-08-01T15:44:19Z)
A Comprehensive Review of AI-based Intelligent Tutoring Systems: Applications and Challenges [0.4369550829556578]
We use a systematic literature review method to analyze numerous qualified studies published from 2010 to 2025.<n>The results reveal a complex landscape regarding the effectiveness of ITS, highlighting both advancements and persistent challenges.
arXiv Detail & Related papers (2025-07-25T01:43:07Z)
A Inteligência Artificial Generativa no Ecossistema Acadêmico: Uma Análise de Aplicações, Desafios e Oportunidades para a Pesquisa, o Ensino e a Divulgação Científica [0.0]
The rapid and disruptive integration of Generative Artificial Intelligence in higher education is reshaping fundamental academic practices.<n>Main challenges include threats to academic integrity, the risk of algorithmic bias, and the need for robust AI literacy.<n>The future of academia will not be defined by resistance to this technology, but by the ability of institutions and individuals to engage with it critically, ethically, and creatively.
arXiv Detail & Related papers (2025-07-03T18:23:18Z)
The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
arXiv Detail & Related papers (2025-06-09T18:37:14Z)
AI-Driven Automation Can Become the Foundation of Next-Era Science of Science Research [58.944125758758936]
The Science of Science (SoS) explores the mechanisms underlying scientific discovery.<n>The advent of artificial intelligence (AI) presents a transformative opportunity for the next generation of SoS.<n>We outline the advantages of AI over traditional methods, discuss potential limitations, and propose pathways to overcome them.
arXiv Detail & Related papers (2025-05-17T15:01:33Z)
Advancing Education through Tutoring Systems: A Systematic Literature Review [3.276010440333338]
This study systematically reviews the transformative role of Tutoring Systems, encompassing Intelligent Tutoring Systems (ITS) and Robot Tutoring Systems (RTS)<n>The findings reveal significant advancements in AI techniques that enhance adaptability, engagement, and learning outcomes.<n>The study highlights the complementary strengths of ITS and RTS, proposing integrated hybrid solutions to maximize educational benefits.
arXiv Detail & Related papers (2025-03-12T18:47:07Z)
Generative AI and Its Impact on Personalized Intelligent Tutoring Systems [0.0]
Generative AI enables personalized education through dynamic content generation, real-time feedback, and adaptive learning pathways. Report explores key applications such as automated question generation, customized feedback mechanisms, and interactive dialogue systems. Future directions highlight the potential advancements in multimodal AI integration, emotional intelligence in tutoring systems, and the ethical implications of AI-driven education.
arXiv Detail & Related papers (2024-10-14T16:01:01Z)
Position Paper: Agent AI Towards a Holistic Intelligence [53.35971598180146]
We emphasize developing Agent AI -- an embodied system that integrates large foundation models into agent actions. In this paper, we propose a novel large action model to achieve embodied intelligent behavior, the Agent Foundation Model.
arXiv Detail & Related papers (2024-02-28T16:09:56Z)
The Future of Fundamental Science Led by Generative Closed-Loop Artificial Intelligence [67.70415658080121]
Recent advances in machine learning and AI are disrupting technological innovation, product development, and society as a whole. AI has contributed less to fundamental science in part because large data sets of high-quality data for scientific practice and model discovery are more difficult to access. Here we explore and investigate aspects of an AI-driven, automated, closed-loop approach to scientific discovery.
arXiv Detail & Related papers (2023-07-09T21:16:56Z)
An interdisciplinary conceptual study of Artificial Intelligence (AI) for helping benefit-risk assessment practices: Towards a comprehensive qualification matrix of AI programs and devices (pre-print 2020) [55.41644538483948]
This paper proposes a comprehensive analysis of existing concepts coming from different disciplines tackling the notion of intelligence. The aim is to identify shared notions or discrepancies to consider for qualifying AI systems.
arXiv Detail & Related papers (2021-05-07T12:01:31Z)
Distributed and Democratized Learning: Philosophy and Research Challenges [80.39805582015133]
We propose a novel design philosophy called democratized learning (Dem-AI) Inspired by the societal groups of humans, the specialized groups of learning agents in the proposed Dem-AI system are self-organized in a hierarchical structure to collectively perform learning tasks more efficiently. We present a reference design as a guideline to realize future Dem-AI systems, inspired by various interdisciplinary fields.
arXiv Detail & Related papers (2020-03-18T08:45:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.