Experimental Evidence on Negative Impact of Generative AI on Scientific Learning Outcomes

• URL: http://arxiv.org/abs/2311.05629v1
• Date: Sat, 23 Sep 2023 21:59:40 GMT
• Title: Experimental Evidence on Negative Impact of Generative AI on Scientific Learning Outcomes
• Authors: Qirui Ju
• Abstract summary: Using AI for summarization significantly improved both quality and output. Individuals with a robust background in the reading topic and superior reading/writing skills benefitted the most.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this study, I explored the impact of Generative AI on learning efficacy in academic reading materials using experimental methods. College-educated participants engaged in three cycles of reading and writing tasks. After each cycle, they responded to comprehension questions related to the material. After adjusting for background knowledge and demographic factors, complete reliance on AI for writing tasks led to a 25.1% reduction in accuracy. In contrast, AI-assisted reading resulted in a 12% decline. Interestingly, using AI for summarization significantly improved both quality and output. Accuracy exhibited notable variance in the AI-assisted section. Further analysis revealed that individuals with a robust background in the reading topic and superior reading/writing skills benefitted the most. I conclude the research by discussing educational policy implications, emphasizing the need for educators to warn students about the dangers of over-dependence on AI and provide guidance on its optimal use in educational settings.

Related papers

• Harnessing AI for efficient analysis of complex policy documents: a case study of Executive Order 14110 [44.99833362998488]
  Policy documents, such as legislation, regulations, and executive orders, are crucial in shaping society. This study aims to evaluate the potential of AI in streamlining policy analysis and to identify the strengths and limitations of current AI approaches.
  arXiv  Detail & Related papers  (2024-06-10T11:19:28Z)
• Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors [2.217351976766501]
  This study delves into university instructors' experiences and attitudes toward AI language models. We find no correlation between teaching style and attitude toward generative AI. While CS educators show far more confidence in their technical understanding of generative AI tools, they show no more confidence in their ability to detect AI-generated work.
  arXiv  Detail & Related papers  (2024-03-22T19:21:29Z)
• Evaluating and Optimizing Educational Content with Large Language Model Judgments [52.33701672559594]
  We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
  arXiv  Detail & Related papers  (2024-03-05T09:09:15Z)
• Generative AI in Writing Research Papers: A New Type of Algorithmic Bias and Uncertainty in Scholarly Work [0.38850145898707145]
  Large language models (LLMs) and generative AI tools present challenges in identifying and addressing biases. generative AI tools are susceptible to goal misgeneralization, hallucinations, and adversarial attacks such as red teaming prompts. We find that incorporating generative AI in the process of writing research manuscripts introduces a new type of context-induced algorithmic bias.
  arXiv  Detail & Related papers  (2023-12-04T04:05:04Z)
• Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making [5.381004207943597]
  We conducted an experiment with seven therapists and ten laypersons on the task of assessing post-stroke survivors' quality of motion. We analyzed their performance, agreement level on the task, and reliance on AI without and with two types of AI explanations. Our work discusses the potential of counterfactual explanations to better estimate the accuracy of an AI model and reduce over-reliance on wrong' AI outputs.
  arXiv  Detail & Related papers  (2023-08-08T16:23:46Z)
• Improving Primary Healthcare Workflow Using Extreme Summarization of Scientific Literature Based on Generative AI [8.901148687545103]
  Our objective is to investigate the potential of generative artificial intelligence in diminishing the cognitive load experienced by practitioners. Our research demonstrates that the use of generative AI for literature review is efficient and effective.
  arXiv  Detail & Related papers  (2023-07-24T21:42:27Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)
• An Exploration of Post-Editing Effectiveness in Text Summarization [58.99765574294715]
  "Post-editing" AI-generated text reduces human workload and improves the quality of AI output. We compare post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience. This study sheds valuable insights on when post-editing is useful for text summarization.
  arXiv  Detail & Related papers  (2022-06-13T18:00:02Z)
• AI Explainability 360: Impact and Design [120.95633114160688]
  In 2019, we created AI Explainability 360 (Arya et al. 2020), an open source software toolkit featuring ten diverse and state-of-the-art explainability methods. This paper examines the impact of the toolkit with several case studies, statistics, and community feedback. The paper also describes the flexible design of the toolkit, examples of its use, and the significant educational material and documentation available to its users.
  arXiv  Detail & Related papers  (2021-09-24T19:17:09Z)
• Personalized Education in the AI Era: What to Expect Next? [76.37000521334585]
  The objective of personalized learning is to design an effective knowledge acquisition track that matches the learner's strengths and bypasses her weaknesses to meet her desired goal. In recent years, the boost of artificial intelligence (AI) and machine learning (ML) has unfolded novel perspectives to enhance personalized education.
  arXiv  Detail & Related papers  (2021-01-19T12:23:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.