LLM-Generated Feedback Supports Learning If Learners Choose to Use It

• URL: http://arxiv.org/abs/2506.17006v1
• Date: Fri, 20 Jun 2025 13:59:14 GMT
• Title: LLM-Generated Feedback Supports Learning If Learners Choose to Use It
• Authors: Danielle R. Thomas, Conrad Borchers, Shambhavi Bhushan, Erin Gatz, Shivang Gupta, Kenneth R. Koedinger,
• Abstract summary: Large language models (LLMs) are increasingly used to generate feedback, yet their impact on learning remains underexplored.<n>This study investigates how on-demand LLM explanatory feedback influences learning in seven scenario-based tutor training lessons.
• Score: 1.4843690728082002
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used to generate feedback, yet their impact on learning remains underexplored, especially compared to existing feedback methods. This study investigates how on-demand LLM-generated explanatory feedback influences learning in seven scenario-based tutor training lessons. Analyzing over 2,600 lesson completions from 885 tutor learners, we compare posttest performance among learners across three groups: learners who received feedback generated by gpt-3.5-turbo, those who declined it, and those without access. All groups received non-LLM corrective feedback. To address potential selection bias-where higher-performing learners may be more inclined to use LLM feedback-we applied propensity scoring. Learners with a higher predicted likelihood of engaging with LLM feedback scored significantly higher at posttest than those with lower propensity. After adjusting for this effect, two out of seven lessons showed statistically significant learning benefits from LLM feedback with standardized effect sizes of 0.28 and 0.33. These moderate effects suggest that the effectiveness of LLM feedback depends on the learners' tendency to seek support. Importantly, LLM feedback did not significantly increase completion time, and learners overwhelmingly rated it as helpful. These findings highlight LLM feedback's potential as a low-cost and scalable way to improve learning on open-ended tasks, particularly in existing systems already providing feedback without LLMs. This work contributes open datasets, LLM prompts, and rubrics to support reproducibility.

Related papers

• Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks [18.695435335031355]
  In subjective annotation tasks with multiple plausible answers, reviewing LLM outputs can change the label distribution.<n>We conducted a pre-registered experiment with 410 unique annotators and over 7,000 annotations.<n>We find that presenting crowdworkers with LLM-generated annotation suggestions did not make them faster, but did improve their self-reported confidence in the task.
  arXiv  Detail & Related papers  (2025-07-21T17:29:21Z)
• Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Study [0.0]
  Large Language Models (LLMs) hold promise as dynamic instructional aids.<n>Yet, it remains unclear whether LLMs can replicate the adaptivity of intelligent tutoring systems (ITS)
  arXiv  Detail & Related papers  (2025-04-07T23:57:32Z)
• Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filtering [66.5524727179286]
  NOVA is a framework designed to identify high-quality data that aligns well with the learned knowledge to reduce hallucinations.<n>It includes Internal Consistency Probing (ICP) and Semantic Equivalence Identification (SEI) to measure how familiar the LLM is with instruction data.<n>To ensure the quality of selected samples, we introduce an expert-aligned reward model, considering characteristics beyond just familiarity.
  arXiv  Detail & Related papers  (2025-02-11T08:05:56Z)
• Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented In-Context Learning for Long-Tail Knowledge in LLMs [50.29035873837]
  Large language models (LLMs) can learn vast amounts of knowledge from diverse domains during pre-training.<n>Long-tail knowledge from specialized domains is often scarce and underrepresented, rarely appearing in the models' memorization.<n>We propose a reinforcement learning-based dynamic uncertainty ranking method for ICL that accounts for the varying impact of each retrieved sample on LLM predictions.
  arXiv  Detail & Related papers  (2024-10-31T03:42:17Z)
• AI Meets the Classroom: When Do Large Language Models Harm Learning? [0.0]
  We show that the effect of large language models (LLMs) on learning outcomes depends on usage behavior.<n>While LLMs show great potential to improve learning, their use must be tailored to the educational context.
  arXiv  Detail & Related papers  (2024-08-29T17:07:46Z)
• Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms [7.550701021850185]
  We investigate the potential of Large Language Models (LLMs) to help students engage in post-lesson reflection. We conducted two randomized field experiments in undergraduate computer science courses.
  arXiv  Detail & Related papers  (2024-06-01T02:41:59Z)
• Improve Temporal Awareness of LLMs for Sequential Recommendation [61.723928508200196]
  Large language models (LLMs) have demonstrated impressive zero-shot abilities in solving a wide range of general-purpose tasks. LLMs fall short in recognizing and utilizing temporal information, rendering poor performance in tasks that require an understanding of sequential data. We propose three prompting strategies to exploit temporal information within historical interactions for LLM-based sequential recommendation.
  arXiv  Detail & Related papers  (2024-05-05T00:21:26Z)
• Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs [6.090496490133132]
  We propose Reinforcement Learning from Multi-role Debates as Feedback (RLDF), a novel approach for bias mitigation replacing human feedback in traditional RLHF. We utilize LLMs in multi-role debates to create a dataset that includes both high-bias and low-bias instances for training the reward model in reinforcement learning.
  arXiv  Detail & Related papers  (2024-04-15T22:18:50Z)
• When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation [66.01754585188739]
  Large Language Models (LLMs) have been found to have difficulty knowing they do not possess certain knowledge. Retrieval Augmentation (RA) has been extensively studied to mitigate LLMs' hallucinations. We propose several methods to enhance LLMs' perception of knowledge boundaries and show that they are effective in reducing overconfidence.
  arXiv  Detail & Related papers  (2024-02-18T04:57:19Z)
• Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning [79.32236399694077]
  Low-quality data in the training set are usually detrimental to instruction tuning. We propose a novel method, termed "reflection-tuning" This approach utilizes an oracle LLM to recycle the original training data by introspecting and enhancing the quality of instructions and responses in the data.
  arXiv  Detail & Related papers  (2023-10-18T05:13:47Z)
• Interpreting Learned Feedback Patterns in Large Language Models [11.601799960959214]
  We train probes to estimate the feedback signal implicit in the activations of a fine-tuned language model. We compare these estimates to the true feedback, measuring how accurate the LFPs are to the fine-tuning feedback. We validate our probes by comparing the neural features they correlate with positive feedback inputs against the features GPT-4 describes and classifies as related to LFPs.
  arXiv  Detail & Related papers  (2023-10-12T09:36:03Z)
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [52.734140807634624]
  Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs. We introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs.
  arXiv  Detail & Related papers  (2023-10-10T16:38:49Z)
• On Learning to Summarize with Large Language Models as References [101.79795027550959]
  Large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. We study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved.
  arXiv  Detail & Related papers  (2023-05-23T16:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.