Evaluating and Optimizing Educational Content with Large Language Model Judgments

• URL: http://arxiv.org/abs/2403.02795v2
• Date: Mon, 6 May 2024 04:54:19 GMT
• Title: Evaluating and Optimizing Educational Content with Large Language Model Judgments
• Authors: Joy He-Yueya, Noah D. Goodman, Emma Brunskill,
• Abstract summary: We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
• Score: 52.33701672559594
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Creating effective educational materials generally requires expensive and time-consuming studies of student learning outcomes. To overcome this barrier, one idea is to build computational models of student learning and use them to optimize instructional materials. However, it is difficult to model the cognitive processes of learning dynamics. We propose an alternative approach that uses Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. Specifically, we use GPT-3.5 to evaluate the overall effect of instructional materials on different student groups and find that it can replicate well-established educational findings such as the Expertise Reversal Effect and the Variability Effect. This demonstrates the potential of LMs as reliable evaluators of educational content. Building on this insight, we introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. We apply this approach to create math word problem worksheets aimed at maximizing student learning gains. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences. We conclude by discussing potential divergences between human and LM opinions and the resulting pitfalls of automating instructional design.

Related papers

• Automated Feedback in Math Education: A Comparative Analysis of LLMs for Open-Ended Responses [0.0]
  This study aims to explore the potential of Large Language Models (LLMs) in facilitating automated feedback in math education. We employ Mistral, a version of Llama catered to math, and fine-tune this model for evaluating student responses by leveraging a dataset of student responses and teacher-written feedback for middle-school math problems. We evaluate the model's performance in scoring accuracy and the quality of feedback by utilizing judgments from 2 teachers.
  arXiv  Detail & Related papers  (2024-10-29T16:57:45Z)
• Students Rather Than Experts: A New AI For Education Pipeline To Model More Human-Like And Personalised Early Adolescences [11.576679362717478]
  This study focuses on language learning as a context for modeling virtual student agents. By curating a dataset of personalized teacher-student interactions with various personality traits, we conduct multi-dimensional evaluation experiments.
  arXiv  Detail & Related papers  (2024-10-21T07:18:24Z)
• LLM-based Cognitive Models of Students with Misconceptions [55.29525439159345]
  This paper investigates whether Large Language Models (LLMs) can be instruction-tuned to meet this dual requirement. We introduce MalAlgoPy, a novel Python library that generates datasets reflecting authentic student solution patterns. Our insights enhance our understanding of AI-based student models and pave the way for effective adaptive learning systems.
  arXiv  Detail & Related papers  (2024-10-16T06:51:09Z)
• Closing the Loop: Learning to Generate Writing Feedback via Language Model Simulated Student Revisions [6.216542656489173]
  We propose PROF that PROduces Feedback via learning from LM simulated student revisions. We empirically test the efficacy of PROF and observe that our approach surpasses a variety of baseline methods in effectiveness of improving students' writing.
  arXiv  Detail & Related papers  (2024-10-10T15:52:48Z)
• Toward In-Context Teaching: Adapting Examples to Students' Misconceptions [54.82965010592045]
  We introduce a suite of models and evaluation methods we call AdapT. AToM is a new probabilistic model for adaptive teaching that jointly infers students' past beliefs and optimize for the correctness of future beliefs. Our results highlight both the difficulty of the adaptive teaching task and the potential of learned adaptive models for solving it.
  arXiv  Detail & Related papers  (2024-05-07T17:05:27Z)
• Improving the Validity of Automatically Generated Feedback via Reinforcement Learning [50.067342343957876]
  We propose a framework for feedback generation that optimize both correctness and alignment using reinforcement learning (RL) Specifically, we use GPT-4's annotations to create preferences over feedback pairs in an augmented dataset for training via direct preference optimization (DPO)
  arXiv  Detail & Related papers  (2024-03-02T20:25:50Z)
• Democratizing Reasoning Ability: Tailored Learning from Large Language Model [97.4921006089966]
  We propose a tailored learning approach to distill such reasoning ability to smaller LMs. We exploit the potential of LLM as a reasoning teacher by building an interactive multi-round learning paradigm. To exploit the reasoning potential of the smaller LM, we propose self-reflection learning to motivate the student to learn from self-made mistakes.
  arXiv  Detail & Related papers  (2023-10-20T07:50:10Z)
• Learning by Self-Explaining [23.420673675343266]
  We introduce a novel workflow in the context of image classification, termed Learning by Self-Explaining (LSX) LSX utilizes aspects of self-refining AI and human-guided explanatory machine learning. Our results indicate improvements via Learning by Self-Explaining on several levels.
  arXiv  Detail & Related papers  (2023-09-15T13:41:57Z)
• Self-directed Machine Learning [86.3709575146414]
  In education science, self-directed learning has been shown to be more effective than passive teacher-guided learning. We introduce the principal concept of Self-directed Machine Learning (SDML) and propose a framework for SDML. Our proposed SDML process benefits from self task selection, self data selection, self model selection, self optimization strategy selection and self evaluation metric selection.
  arXiv  Detail & Related papers  (2022-01-04T18:32:06Z)
• RLTutor: Reinforcement Learning Based Adaptive Tutoring System by Modeling Virtual Student with Fewer Interactions [10.34673089426247]
  We propose a framework for optimizing teaching strategies by constructing a virtual model of the student. Our results can serve as a buffer between theoretical instructional optimization and practical applications in e-learning systems.
  arXiv  Detail & Related papers  (2021-07-31T15:42:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.