AI Meets the Classroom: When Does ChatGPT Harm Learning?

• URL: http://arxiv.org/abs/2409.09047v1
• Date: Thu, 29 Aug 2024 17:07:46 GMT
• Title: AI Meets the Classroom: When Does ChatGPT Harm Learning?
• Authors: Matthias Lehmann, Philipp B. Cornelius, Fabian J. Sting,
• Abstract summary: We study how generative AI and specifically large language models (LLMs) impact learning in coding classes. We show across three studies that LLM usage can have positive and negative effects on learning outcomes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we study how generative AI and specifically large language models (LLMs) impact learning in coding classes. We show across three studies that LLM usage can have positive and negative effects on learning outcomes. Using observational data from university-level programming courses, we establish such effects in the field. We replicate these findings in subsequent experimental studies, which closely resemble typical learning scenarios, to show causality. We find evidence for two contrasting mechanisms that determine the overall effect of LLM usage on learning. Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to solve practice exercises for them and thus do not invest sufficient own mental effort. Those who never used LLMs before are particularly prone to such adverse behavior. Students without prior domain knowledge gain more from having access to LLMs. Finally, we show that the self-perceived benefits of using LLMs for learning exceed the actual benefits, potentially resulting in an overestimation of one's own abilities. Overall, our findings show promising potential of LLMs as learning support, however also that students have to be very cautious of possible pitfalls.

Related papers

• Refine Knowledge of Large Language Models via Adaptive Contrastive Learning [54.61213933999464]
  A mainstream category of methods is to reduce hallucinations by optimizing the knowledge representation of Large Language Models. We believe that the process of models refining knowledge can greatly benefit from the way humans learn. In our work, by imitating the human learning process, we design an Adaptive Contrastive Learning strategy.
  arXiv  Detail & Related papers  (2025-02-11T02:19:13Z)
• Embracing AI in Education: Understanding the Surge in Large Language Model Use by Secondary Students [53.20318273452059]
  Large language models (LLMs) like OpenAI's ChatGPT have opened up new avenues in education. Despite school restrictions, our survey of over 300 middle and high school students revealed that a remarkable 70% of students have utilized LLMs. We propose a few ideas to address such issues, including subject-specific models, personalized learning, and AI classrooms.
  arXiv  Detail & Related papers  (2024-11-27T19:19:34Z)
• Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course [0.9718746651638346]
  Large language models (LLMs) can generate, debug, and explain code. Our study explores how students' social perceptions influence their own LLM usage.
  arXiv  Detail & Related papers  (2024-06-10T16:40:14Z)
• Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs [6.090496490133132]
  We propose Reinforcement Learning from Multi-role Debates as Feedback (RLDF), a novel approach for bias mitigation replacing human feedback in traditional RLHF. We utilize LLMs in multi-role debates to create a dataset that includes both high-bias and low-bias instances for training the reward model in reinforcement learning.
  arXiv  Detail & Related papers  (2024-04-15T22:18:50Z)
• An Exploratory Study on Upper-Level Computing Students' Use of Large Language Models as Tools in a Semester-Long Project [2.7325338323814328]
  The purpose of this study is to explore computing students' experiences and approaches to using LLMs during a semester-long software engineering project. We collected data from a senior-level software engineering course at Purdue University. We analyzed the data to identify themes related to students' usage patterns and learning outcomes.
  arXiv  Detail & Related papers  (2024-03-27T15:21:58Z)
• Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs [60.40396361115776]
  This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in large language models (LLMs) with a slim proxy model. We employ a proxy model which has far fewer parameters, and take its answers as answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM.
  arXiv  Detail & Related papers  (2024-02-19T11:11:08Z)
• When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation [66.01754585188739]
  Large Language Models (LLMs) have been found to have difficulty knowing they do not possess certain knowledge. Retrieval Augmentation (RA) has been extensively studied to mitigate LLMs' hallucinations. We propose several methods to enhance LLMs' perception of knowledge boundaries and show that they are effective in reducing overconfidence.
  arXiv  Detail & Related papers  (2024-02-18T04:57:19Z)
• Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation [23.736611338497244]
  TinyLLM is a new knowledge distillation paradigm to learn a small student LLM from multiple large teacher LLMs. We introduce an in-context example generator and a teacher-forcing Chain-of-Thought strategy to ensure that the rationales are accurate and grounded in contextually appropriate scenarios. Results show that TinyLLM can outperform large teacher LLMs significantly, despite a considerably smaller model size.
  arXiv  Detail & Related papers  (2024-02-07T06:48:24Z)
• See the Unseen: Better Context-Consistent Knowledge-Editing by Noises [73.54237379082795]
  Knowledge-editing updates knowledge of large language models (LLMs) Existing works ignore this property and the editing lacks generalization. We empirically find that the effects of different contexts upon LLMs in recalling the same knowledge follow a Gaussian-like distribution.
  arXiv  Detail & Related papers  (2024-01-15T09:09:14Z)
• Enabling Large Language Models to Learn from Rules [99.16680531261987]
  We are inspired that humans can learn the new tasks or knowledge in another way by learning from rules. We propose rule distillation, which first uses the strong in-context abilities of LLMs to extract the knowledge from the textual rules. Our experiments show that making LLMs learn from rules by our method is much more efficient than example-based learning in both the sample size and generalization ability.
  arXiv  Detail & Related papers  (2023-11-15T11:42:41Z)
• Making Harmful Behaviors Unlearnable for Large Language Models [50.44915524846857]
  Large language models (LLMs) have shown great potential as general-purpose AI assistants in various domains. LLMs can be easily fine-tuned into harmful assistants as the fine-tuning data often contains implicit or explicit harmful content. This paper proposes a controllable training framework that makes harmful behaviors unlearnable during the fine-tuning process.
  arXiv  Detail & Related papers  (2023-11-02T09:18:21Z)
• Democratizing Reasoning Ability: Tailored Learning from Large Language Model [97.4921006089966]
  We propose a tailored learning approach to distill such reasoning ability to smaller LMs. We exploit the potential of LLM as a reasoning teacher by building an interactive multi-round learning paradigm. To exploit the reasoning potential of the smaller LM, we propose self-reflection learning to motivate the student to learn from self-made mistakes.
  arXiv  Detail & Related papers  (2023-10-20T07:50:10Z)
• Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perception [19.335003380399527]
  Large language models (LLMs) offer a promising avenue, with increasing research exploring their educational utility. Our work highlights the role that teachers can play in shaping LLM-supported learning environments.
  arXiv  Detail & Related papers  (2023-10-13T01:21:52Z)
• Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation [109.8527403904657]
  We show that large language models (LLMs) possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We propose a simple method to dynamically utilize supporting documents with our judgement strategy.
  arXiv  Detail & Related papers  (2023-07-20T16:46:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.