Experimental Evidence on Negative Impact of Generative AI on Scientific
Learning Outcomes
- URL: http://arxiv.org/abs/2311.05629v1
- Date: Sat, 23 Sep 2023 21:59:40 GMT
- Title: Experimental Evidence on Negative Impact of Generative AI on Scientific
Learning Outcomes
- Authors: Qirui Ju
- Abstract summary: Using AI for summarization significantly improved both quality and output.
Individuals with a robust background in the reading topic and superior reading/writing skills benefitted the most.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this study, I explored the impact of Generative AI on learning efficacy in
academic reading materials using experimental methods. College-educated
participants engaged in three cycles of reading and writing tasks. After each
cycle, they responded to comprehension questions related to the material. After
adjusting for background knowledge and demographic factors, complete reliance
on AI for writing tasks led to a 25.1% reduction in accuracy. In contrast,
AI-assisted reading resulted in a 12% decline. Interestingly, using AI for
summarization significantly improved both quality and output. Accuracy
exhibited notable variance in the AI-assisted section. Further analysis
revealed that individuals with a robust background in the reading topic and
superior reading/writing skills benefitted the most. I conclude the research by
discussing educational policy implications, emphasizing the need for educators
to warn students about the dangers of over-dependence on AI and provide
guidance on its optimal use in educational settings.
Related papers
- Harnessing AI for efficient analysis of complex policy documents: a case study of Executive Order 14110 [44.99833362998488]
Policy documents, such as legislation, regulations, and executive orders, are crucial in shaping society.
This study aims to evaluate the potential of AI in streamlining policy analysis and to identify the strengths and limitations of current AI approaches.
arXiv Detail & Related papers (2024-06-10T11:19:28Z) - Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors [2.217351976766501]
This study delves into university instructors' experiences and attitudes toward AI language models.
We find no correlation between teaching style and attitude toward generative AI.
While CS educators show far more confidence in their technical understanding of generative AI tools, they show no more confidence in their ability to detect AI-generated work.
arXiv Detail & Related papers (2024-03-22T19:21:29Z) - Evaluating and Optimizing Educational Content with Large Language Model Judgments [52.33701672559594]
We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes.
We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function.
Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
arXiv Detail & Related papers (2024-03-05T09:09:15Z) - Generative AI in Writing Research Papers: A New Type of Algorithmic Bias
and Uncertainty in Scholarly Work [0.38850145898707145]
Large language models (LLMs) and generative AI tools present challenges in identifying and addressing biases.
generative AI tools are susceptible to goal misgeneralization, hallucinations, and adversarial attacks such as red teaming prompts.
We find that incorporating generative AI in the process of writing research manuscripts introduces a new type of context-induced algorithmic bias.
arXiv Detail & Related papers (2023-12-04T04:05:04Z) - Understanding the Effect of Counterfactual Explanations on Trust and
Reliance on AI for Human-AI Collaborative Clinical Decision Making [5.381004207943597]
We conducted an experiment with seven therapists and ten laypersons on the task of assessing post-stroke survivors' quality of motion.
We analyzed their performance, agreement level on the task, and reliance on AI without and with two types of AI explanations.
Our work discusses the potential of counterfactual explanations to better estimate the accuracy of an AI model and reduce over-reliance on wrong' AI outputs.
arXiv Detail & Related papers (2023-08-08T16:23:46Z) - Improving Primary Healthcare Workflow Using Extreme Summarization of
Scientific Literature Based on Generative AI [8.901148687545103]
Our objective is to investigate the potential of generative artificial intelligence in diminishing the cognitive load experienced by practitioners.
Our research demonstrates that the use of generative AI for literature review is efficient and effective.
arXiv Detail & Related papers (2023-07-24T21:42:27Z) - The Role of AI in Drug Discovery: Challenges, Opportunities, and
Strategies [97.5153823429076]
The benefits, challenges and drawbacks of AI in this field are reviewed.
The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
arXiv Detail & Related papers (2022-12-08T23:23:39Z) - An Exploration of Post-Editing Effectiveness in Text Summarization [58.99765574294715]
"Post-editing" AI-generated text reduces human workload and improves the quality of AI output.
We compare post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience.
This study sheds valuable insights on when post-editing is useful for text summarization.
arXiv Detail & Related papers (2022-06-13T18:00:02Z) - AI Explainability 360: Impact and Design [120.95633114160688]
In 2019, we created AI Explainability 360 (Arya et al. 2020), an open source software toolkit featuring ten diverse and state-of-the-art explainability methods.
This paper examines the impact of the toolkit with several case studies, statistics, and community feedback.
The paper also describes the flexible design of the toolkit, examples of its use, and the significant educational material and documentation available to its users.
arXiv Detail & Related papers (2021-09-24T19:17:09Z) - Personalized Education in the AI Era: What to Expect Next? [76.37000521334585]
The objective of personalized learning is to design an effective knowledge acquisition track that matches the learner's strengths and bypasses her weaknesses to meet her desired goal.
In recent years, the boost of artificial intelligence (AI) and machine learning (ML) has unfolded novel perspectives to enhance personalized education.
arXiv Detail & Related papers (2021-01-19T12:23:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.