Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach

• URL: http://arxiv.org/abs/2407.12687v2
• Date: Fri, 19 Jul 2024 14:03:41 GMT
• Title: Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach
• Authors: Irina Jurenka, Markus Kunesch, Kevin R. McKee, Daniel Gillick, Shaojian Zhu, Sara Wiltberger, Shubham Milind Phal, Katherine Hermann, Daniel Kasenberg, Avishkar Bhoopchand, Ankit Anand, Miruna Pîslar, Stephanie Chan, Lisa Wang, Jennifer She, Parsa Mahmoudieh, Aliya Rysbek, Wei-Jen Ko, Andrea Huber, Brett Wiltshire, Gal Elidan, Roni Rabin, Jasmin Rubinovitz, Amit Pitaru, Mac McAllister, Julia Wilkowski, David Choi, Roee Engelberg, Lidan Hackmon, Adva Levin, Rachel Griffin, Michael Sears, Filip Bar, Mia Mesar, Mana Jabbour, Arslan Chaudhry, James Cohan, Sridhar Thiagarajan, Nir Levine, Ben Brown, Dilan Gorur, Svetlana Grant, Rachel Hashimshoni, Laura Weidinger, Jieru Hu, Dawn Chen, Kuba Dolecki, Canfer Akbulut, Maxwell Bileschi, Laura Culp, Wen-Xin Dong, Nahema Marchal, Kelsie Van Deman, Hema Bajaj Misra, Michael Duah, Moran Ambar, Avi Caciularu, Sandra Lefdal, Chris Summerfield, James An, Pierre-Alexandre Kamienny, Abhinit Mohdi, Theofilos Strinopoulous, Annie Hale, Wayne Anderson, Luis C. Cobo, Niv Efron, Muktha Ananda, Shakir Mohamed, Maureen Heymans, Zoubin Ghahramani, Yossi Matias, Ben Gomes, Lila Ibrahim,
• Abstract summary: Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks.
• Score: 25.903775277417267
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A major challenge facing the world is the provision of equitable and universal access to quality education. Recent advances in generative AI (gen AI) have created excitement about the potential of new technologies to offer a personal tutor for every learner and a teaching assistant for every teacher. The full extent of this dream, however, has not yet materialised. We argue that this is primarily due to the difficulties with verbalising pedagogical intuitions into gen AI prompts and the lack of good evaluation practices, reinforced by the challenges in defining excellent pedagogy. Here we present our work collaborating with learners and educators to translate high level principles from learning science into a pragmatic set of seven diverse educational benchmarks, spanning quantitative, qualitative, automatic and human evaluations; and to develop a new set of fine-tuning datasets to improve the pedagogical capabilities of Gemini, introducing LearnLM-Tutor. Our evaluations show that LearnLM-Tutor is consistently preferred over a prompt tuned Gemini by educators and learners on a number of pedagogical dimensions. We hope that this work can serve as a first step towards developing a comprehensive educational evaluation framework, and that this can enable rapid progress within the AI and EdTech communities towards maximising the positive impact of gen AI in education.

Related papers

• Bidirectional Human-AI Alignment in Education for Trustworthy Learning Environments [7.0064528229443]
  Artificial intelligence (AI) is transforming education, offering unprecedented opportunities to personalize learning, enhance assessment, and support educators.<n>Yet these opportunities also introduce risks related to equity, privacy, and student autonomy.<n>This chapter develops the concept of bidirectional human-AI alignment in education, emphasizing that trustworthy learning environments arise not only from embedding human values into AI systems but also from equipping teachers, students, and institutions with the skills to interpret, critique, and guide these technologies.
  arXiv  Detail & Related papers  (2025-12-25T07:50:56Z)
• Towards Synergistic Teacher-AI Interactions with Generative Artificial Intelligence [4.647571398484235]
  Automation of teaching tasks through GenAI raises concerns about reduced teacher agency, potential cognitive atrophy, and the broader deprofessionalisation of teaching.<n>This chapter presents a conceptualisation of five levels of teacher-AI teaming: transactional, situational, operational, praxical and synergistic teaming.<n>We outline a future vision that moves beyond individual teacher agency toward collaborative decision-making between teachers and AI.
  arXiv  Detail & Related papers  (2025-11-24T18:29:29Z)
• Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems [15.954407353419258]
  generative AI (GenAI) models have accelerated the development of large language model (LLM)-powered Intelligent Tutoring Systems (ITSs)<n>However, the progress and impact of these systems remain largely untraceable due to the absence of reliable, universally accepted, and pedagogy-driven evaluation frameworks and benchmarks.<n>Most existing educational dialogue-based ITS evaluations rely on subjective protocols and non-standardized benchmarks, leading to inconsistencies and limited generalizability.<n>This work provides comprehensive state-of-the-art evaluation practices, highlighting associated challenges through real-world case studies from careful and caring AIED research.
  arXiv  Detail & Related papers  (2025-10-26T08:44:21Z)
• Generative AI in Training and Coaching: Redefining the Design Process of Learning Materials [44.99833362998488]
  We explore how AI integrates into the design process of learning materials, assessing its impact on efficiency, pedagogical quality, and the evolving role of human trainers and coaches.<n>Through qualitative interviews with professionals in education and corporate training, we identify the following key topics.<n>We derive how tools based on GenAI can successfully be implemented for trainers and coaches on an individual, organizational, systemic, and strategic level.
  arXiv  Detail & Related papers  (2025-08-06T03:42:43Z)
• From Recall to Reasoning: Automated Question Generation for Deeper Math Learning through Large Language Models [44.99833362998488]
  We investigated the first steps for optimizing content creation for advanced math.<n>We looked at the ability of GenAI to produce high-quality practice problems that are relevant to the course content.
  arXiv  Detail & Related papers  (2025-05-17T08:30:10Z)
• Generative AI in Education: From Foundational Insights to the Socratic Playground for Learning [2.8947618493306324]
  We discuss parallels between Large Language Models (LLMs) and human cognition. We show how generative AI can drive personalized learning at scale.
  arXiv  Detail & Related papers  (2025-01-12T01:43:39Z)
• Transforming Teacher Education in Developing Countries: The Role of Generative AI in Bridging Theory and Practice [0.7416846035207727]
  The study focuses on Ghana, where challenges such as limited pedagogical modeling, performance-based assessments, and practitioner-expertise gaps hinder progress. GenAI has the capacity to address these issues by supporting content knowledge acquisition, a role that currently dominates teacher education programs. The study concludes by recommending empirical research to explore these roles further and develop practical steps for integrating GenAI into teacher education systems responsibly and effectively.
  arXiv  Detail & Related papers  (2024-11-16T06:46:09Z)
• From MOOC to MAIC: Reshaping Online Teaching and Learning through LLM-driven Agents [78.15899922698631]
  MAIC (Massive AI-empowered Course) is a new form of online education that leverages LLM-driven multi-agent systems to construct an AI-augmented classroom. We conduct preliminary experiments at Tsinghua University, one of China's leading universities.
  arXiv  Detail & Related papers  (2024-09-05T13:22:51Z)
• Collaborative Design of AI-Enhanced Learning Activities [0.0]
  We develop a formative intervention that enables preservice teachers, in-service teachers, and EdTech specialists to effectively incorporate AI into their teaching practices. Participants reflect on AI's potential in teaching and learning by exploring different activities that can integrate AI literacy in education, including its ethical considerations and potential for innovative pedagogy.
  arXiv  Detail & Related papers  (2024-07-09T08:34:08Z)
• Exploring Teachers' Perception of Artificial Intelligence: The Socio-emotional Deficiency as Opportunities and Challenges in Human-AI Complementarity in K-12 Education [1.9797215742507548]
  In schools, teachers play a multitude of roles, serving as educators, counselors, decision-makers, and members of the school community. With recent advances in artificial intelligence (AI), there is increasing discussion about how AI can assist, complement, and collaborate with teachers. Our study seeks educators' perspectives on the potential strengths and limitations of AI across a spectrum of responsibilities.
  arXiv  Detail & Related papers  (2024-05-20T15:43:04Z)
• Enhancing Instructional Quality: Leveraging Computer-Assisted Textual Analysis to Generate In-Depth Insights from Educational Artifacts [13.617709093240231]
  We examine how artificial intelligence (AI) and machine learning (ML) methods can analyze educational content, teacher discourse, and student responses to foster instructional improvement. We identify key areas where AI/ML integration offers significant advantages, including teacher coaching, student support, and content development. This paper emphasizes the importance of aligning AI/ML technologies with pedagogical goals to realize their full potential in educational settings.
  arXiv  Detail & Related papers  (2024-03-06T18:29:18Z)
• Bringing Generative AI to Adaptive Learning in Education [58.690250000579496]
  We shed light on the intersectional studies of generative AI and adaptive learning. We argue that this union will contribute significantly to the development of the next-stage learning format in education.
  arXiv  Detail & Related papers  (2024-02-02T23:54:51Z)
• Generative AI and Its Educational Implications [0.0]
  We discuss the implications of generative AI on education across four critical sections. We propose ways in which generative AI can transform the educational landscape. Acknowledging the societal impact, we emphasize the need for updating curricula.
  arXiv  Detail & Related papers  (2023-12-26T21:29:31Z)
• Exploration with Principles for Diverse AI Supervision [88.61687950039662]
  Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI. While this generative AI approach has produced impressive results, it heavily leans on human supervision. This strong reliance on human oversight poses a significant hurdle to the advancement of AI innovation. We propose a novel paradigm termed Exploratory AI (EAI) aimed at autonomously generating high-quality training data.
  arXiv  Detail & Related papers  (2023-10-13T07:03:39Z)
• An Experience Report of Executive-Level Artificial Intelligence Education in the United Arab Emirates [53.04281982845422]
  We present an experience report of teaching an AI course to business executives in the United Arab Emirates (UAE) Rather than focusing only on theoretical and technical aspects, we developed a course that teaches AI with a view to enabling students to understand how to incorporate it into existing business processes.
  arXiv  Detail & Related papers  (2022-02-02T20:59:53Z)
• Personalized Education in the AI Era: What to Expect Next? [76.37000521334585]
  The objective of personalized learning is to design an effective knowledge acquisition track that matches the learner's strengths and bypasses her weaknesses to meet her desired goal. In recent years, the boost of artificial intelligence (AI) and machine learning (ML) has unfolded novel perspectives to enhance personalized education.
  arXiv  Detail & Related papers  (2021-01-19T12:23:32Z)
• Creation and Evaluation of a Pre-tertiary Artificial Intelligence (AI) Curriculum [58.86139968005518]
  The Chinese University of Hong Kong (CUHK)-Jockey Club AI for the Future Project (AI4Future) co-created an AI curriculum for pre-tertiary education. A team of 14 professors with expertise in engineering and education collaborated with 17 principals and teachers from 6 secondary schools to co-create the curriculum. The co-creation process generated a variety of resources which enhanced the teachers knowledge in AI, as well as fostered teachers autonomy in bringing the subject matter into their classrooms.
  arXiv  Detail & Related papers  (2021-01-19T11:26:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.