Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences

• URL: http://arxiv.org/abs/2405.04645v2
• Date: Sat, 11 May 2024 17:36:58 GMT
• Title: Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences
• Authors: John Stamper, Ruiwei Xiao, Xinying Hou,
• Abstract summary: This work advocates careful and caring AIED research by going through previous research on feedback generation in ITS. The main contributions of this paper include: an avocation of applying more cautious, theoretically grounded methods in feedback generation in the era of generative AI.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The field of Artificial Intelligence in Education (AIED) focuses on the intersection of technology, education, and psychology, placing a strong emphasis on supporting learners' needs with compassion and understanding. The growing prominence of Large Language Models (LLMs) has led to the development of scalable solutions within educational settings, including generating different types of feedback in Intelligent Tutoring Systems. However, the approach to utilizing these models often involves directly formulating prompts to solicit specific information, lacking a solid theoretical foundation for prompt construction and empirical assessments of their impact on learning. This work advocates careful and caring AIED research by going through previous research on feedback generation in ITS, with emphasis on the theoretical frameworks they utilized and the efficacy of the corresponding design in empirical evaluations, and then suggesting opportunities to apply these evidence-based principles to the design, experiment, and evaluation phases of LLM-based feedback generation. The main contributions of this paper include: an avocation of applying more cautious, theoretically grounded methods in feedback generation in the era of generative AI; and practical suggestions on theory and evidence-based feedback design for LLM-powered ITS.

Related papers

• A Novel Psychometrics-Based Approach to Developing Professional Competency Benchmark for Large Language Models [0.0]
  We propose a comprehensive approach to benchmark development based on rigorous psychometric principles. We make the first attempt to illustrate this approach by creating a new benchmark in the field of pedagogy and education. We construct a novel benchmark guided by the Bloom's taxonomy and rigorously designed by a consortium of education experts trained in test development.
  arXiv  Detail & Related papers  (2024-10-29T19:32:43Z)
• Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs [64.9693406713216]
  Internal mechanisms that contribute to the effectiveness of RAG systems remain underexplored. Our experiments reveal that several core groups of experts are primarily responsible for RAG-related behaviors. We propose several strategies to enhance RAG's efficiency and effectiveness through expert activation.
  arXiv  Detail & Related papers  (2024-10-20T16:08:54Z)
• GIVE: Structured Reasoning with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning framework that integrates the parametric and non-parametric memories. Our method facilitates a more logical and step-wise reasoning approach akin to experts' problem-solving, rather than gold answer retrieval.
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• IDEA: Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abduction [3.961279440272764]
  We introduce RULEARN, a novel benchmark designed to assess the rule-learning abilities of large language models in interactive settings. We propose IDEA, a novel reasoning framework that integrates the process of Induction, Deduction, and Abduction. Our evaluation of the IDEA framework, which involves five representative LLMs, demonstrates significant improvements over the baseline.
  arXiv  Detail & Related papers  (2024-08-19T23:37:07Z)
• Exploring the landscape of large language models: Foundations, techniques, and challenges [8.042562891309414]
  The article sheds light on the mechanics of in-context learning and a spectrum of fine-tuning approaches. It explores how LLMs can be more closely aligned with human preferences through innovative reinforcement learning frameworks. The ethical dimensions of LLM deployment are discussed, underscoring the need for mindful and responsible application.
  arXiv  Detail & Related papers  (2024-04-18T08:01:20Z)
• RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs [49.386699863989335]
  Training large language models (LLMs) to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals.
  arXiv  Detail & Related papers  (2024-04-12T15:54:15Z)
• Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus [4.569421189811511]
  We introduce a novel approach to evaluate the inference and contextual understanding abilities of Large Language Models (LLMs) We focus on three key components from the Language of Thought Hypothesis (LoTH): Logical Coherence, Compositionality, and Productivity. Our experiments reveal that while LLMs demonstrate some inference capabilities, they still significantly lag behind human-level reasoning in these three aspects.
  arXiv  Detail & Related papers  (2024-03-18T13:50:50Z)
• Evaluating and Optimizing Educational Content with Large Language Model Judgments [52.33701672559594]
  We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
  arXiv  Detail & Related papers  (2024-03-05T09:09:15Z)
• Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach [50.125704610228254]
  Large Language Models (LLMs) have not only exhibited exceptional performance across various tasks, but also demonstrated sparks of intelligence. Recent studies have focused on assessing their capabilities on human exams and revealed their impressive competence in different domains. We conduct an evaluation using MoocRadar, a meticulously annotated human test dataset based on Bloom taxonomy.
  arXiv  Detail & Related papers  (2023-10-12T09:55:45Z)
• Scaling Evidence-based Instructional Design Expertise through Large Language Models [0.0]
  This paper explores leveraging Large Language Models (LLMs), specifically GPT-4, in the field of instructional design. With a focus on scaling evidence-based instructional design expertise, our research aims to bridge the gap between theoretical educational studies and practical implementation. We discuss the benefits and limitations of AI-driven content generation, emphasizing the necessity of human oversight in ensuring the quality of educational materials.
  arXiv  Detail & Related papers  (2023-05-31T17:54:07Z)
• Towards Interpretable Deep Learning Models for Knowledge Tracing [62.75876617721375]
  We propose to adopt the post-hoc method to tackle the interpretability issue for deep learning based knowledge tracing (DLKT) models. Specifically, we focus on applying the layer-wise relevance propagation (LRP) method to interpret RNN-based DLKT model. Experiment results show the feasibility using the LRP method for interpreting the DLKT model's predictions.
  arXiv  Detail & Related papers  (2020-05-13T04:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.