Short-Term Gains, Long-Term Gaps: The Impact of GenAI and Search Technologies on Retention

• URL: http://arxiv.org/abs/2507.07357v1
• Date: Thu, 10 Jul 2025 00:44:50 GMT
• Title: Short-Term Gains, Long-Term Gaps: The Impact of GenAI and Search Technologies on Retention
• Abstract summary: This study investigates how GenAI (ChatGPT), search engines (Google), and e-textbooks influence student performance across tasks of varying cognitive complexity.<n>ChatGPT and Google groups outperformed the control group in immediate assessments for lower-order cognitive tasks.<n>While AI-driven tools facilitate immediate performance, they do not inherently reinforce long-term retention unless supported by structured learning strategies.
• Score: 1.534667887016089
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rise of Generative AI (GenAI) tools, such as ChatGPT, has transformed how students access and engage with information, raising questions about their impact on learning outcomes and retention. This study investigates how GenAI (ChatGPT), search engines (Google), and e-textbooks influence student performance across tasks of varying cognitive complexity, based on Bloom's Taxonomy. Using a sample of 123 students, we examined performance in three tasks: [1] knowing and understanding, [2] applying, and [3] synthesizing, evaluating, and creating. Results indicate that ChatGPT and Google groups outperformed the control group in immediate assessments for lower-order cognitive tasks, benefiting from quick access to structured information. However, their advantage diminished over time, with retention test scores aligning with those of the e-textbook group. For higher-order cognitive tasks, no significant differences were observed among groups, with the control group demonstrating the highest retention. These findings suggest that while AI-driven tools facilitate immediate performance, they do not inherently reinforce long-term retention unless supported by structured learning strategies. The study highlights the need for balanced technology integration in education, ensuring that AI tools are paired with pedagogical approaches that promote deep cognitive engagement and knowledge retention.

Related papers

• A Survey on Self-Improving Test-Time Intelligence: Feedback-Driven Adapting, Learning, and Scaling at Inference [72.19201465132619]
  We present feedback-driven Test-Time Intelligence (TTI) as a unified perspective for understanding such deployment-time improvement.<n>We use this view to relate test-time adaptation, test-time learning, and test-time scaling, highlighting both their distinctions and their growing overlap in hybrid systems.<n>Our goal is to provide a coherent foundation and research roadmap for the study of self-improving AI systems at test time.
  arXiv  Detail & Related papers  (2026-09-01T11:41:39Z)
• Is Solving Better Than Evaluating GenAI Solutions? [45.714140455691485]
  This study compares evaluating GenAI-generated solutions with traditional problem solving in a junior-level algorithms course.<n>We found no statistically significant differences between groups in midterm scores, final exam scores, overall course grades, or exam problems structurally aligned with the homework interventions.<n>Students received significantly higher homework scores when evaluating GenAI-generated solutions, but this localized advantage did not translate into downstream gains.
  arXiv  Detail & Related papers  (2026-07-30T02:08:29Z)
• Learning by Chatting? Investigating the Impact of Generative AI on Information Seeking and Learning [33.12893683348563]
  Generative AI (GenAI) tools offer opportunities for augmenting human cognitive tasks.<n>Among these tasks, information seeking is rapidly reshaped by GenAI tools, with potentially profound implications for learning and knowledge acquisition.<n>We conducted a field experiment in which participants sought information through either ChatGPT or Google Search over a span of 8 days.<n>Our findings show that participants in the ChatGPT group experienced diminished agency in their information-seeking processes, as they offloaded much of the information selection to AI.
  arXiv  Detail & Related papers  (2026-06-10T05:28:32Z)
• The Role of Instructional Guidance in Generative AI-Assisted Learning: Empirical Evidence from Construction Engineering Education [2.856083627975759]
  This study examines how instructional guidance shapes student and AI interaction in construction education.<n>A five-step prompting framework grounded in Generative Learning Theory is introduced to guide learner interaction.
  arXiv  Detail & Related papers  (2026-06-03T23:27:46Z)
• Patterns of Learner-AI Interaction and Academic Performance in an Object-Oriented Programming Course [0.0]
  This research paper examines how different forms of learner-AI interaction relate to learning outcomes in object-oriented programming (OOP) courses.<n>Survey data were collected from 210 undergraduate students enrolled in a first-year OOP course.
  arXiv  Detail & Related papers  (2026-05-26T07:48:00Z)
• AI Literacy, Safety Awareness, and STEM Career Aspirations of Australian Secondary Students: Evaluating the Impact of Workshop Interventions [38.350232667249095]
  Deepfakes and other forms of synthetic media pose growing safety risks for adolescents.<n>This study evaluates the impact of Day of AI Australia's workshop-based intervention on Australian secondary students.
  arXiv  Detail & Related papers  (2026-01-30T02:55:53Z)
• Socratic Mind: Impact of a Novel GenAI-Powered Assessment Tool on Student Learning and Higher-Order Thinking [2.192176712066146]
  This study examines the impact of Socratic Mind, a Generative Artificial Intelligence (GenAI) powered formative assessment tool on learning outcomes.<n>Students who engaged with the GenAI tool experienced significant gains in their quiz scores compared to those who did not.<n>Our findings highlight the promise of AI-mediated dialogue in fostering deeper engagement and higher-order cognitive skills.
  arXiv  Detail & Related papers  (2025-09-18T03:08:24Z)
• Do Students Rely on AI? Analysis of Student-ChatGPT Conversations from a Field Study [10.71612026319996]
  This study analyzed 315 student-AI conversations during a brief, quiz-based scenario across various STEM courses.<n>Students exhibited overall low reliance on AI and many of them could not effectively use AI for learning.<n>Certain behavioral metrics strongly predicted AI reliance, highlighting potential behavioral mechanisms to explain AI adoption.
  arXiv  Detail & Related papers  (2025-08-27T20:00:27Z)
• ChatGPT produces more "lazy" thinkers: Evidence of cognitive engagement decline [0.0]
  This study investigates the impact of generative artificial intelligence (AI) tools on the cognitive engagement of students during academic writing tasks.<n>The results revealed significantly lower cognitive engagement scores in the ChatGPT group compared to the control group.<n>These findings suggest that AI assistance may lead to cognitive offloading.
  arXiv  Detail & Related papers  (2025-06-30T18:41:50Z)
• Robots and Children that Learn Together : Improving Knowledge Retention by Teaching Peer-Like Interactive Robots [41.94295877935867]
  This study introduces Interactive Reinforcement Learning as a cognitive model for teachable social robots.<n>Children in the LbT condition achieved significantly higher retention gains compared to those in the self-practice condition.<n>This work makes two contributions: (1) it introduces Interactive RL as a pedagogically effective and scalable model for peer-robot learning, and (2) it demonstrates, for the first time, the feasibility of deploying multiple autonomous robots simultaneously in real classrooms.
  arXiv  Detail & Related papers  (2025-06-23T07:51:04Z)
• AI instructional agent improves student's perceived learner control and learning outcome: empirical evidence from a randomized controlled trial [29.45751702212421]
  This study examines the impact of an AI instructional agent on students' perceived learner control and academic performance in a medium demanding course with lecturing as the main teaching strategy.<n>Students in the AI instructional agent group reported significantly higher levels of perceived learner control compared to the other groups.
  arXiv  Detail & Related papers  (2025-05-28T16:13:27Z)
• Evaluating the AI-Lab Intervention: Impact on Student Perception and Use of Generative AI in Early Undergraduate Computer Science Courses [0.0]
  Generative AI (GenAI) is rapidly entering computer science education.<n>Concerns about overreliance coexist with a gap in research on structured scaffolding to guide tool use in formal courses.<n>This study examines the impact of a dedicated "AI-Lab" intervention on undergraduate students.
  arXiv  Detail & Related papers  (2025-04-30T18:12:42Z)
• Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance [15.064064880549232]
  The concept of hybrid intelligence is still at a nascent stage.<n>How learners can benefit from a symbiotic relationship with various agents is still unknown.<n>ChatGPT may promote learners' dependence on technology and potentially trigger metacognitive laziness.
  arXiv  Detail & Related papers  (2024-12-12T14:32:39Z)
• Analyzing the Impact of AI Tools on Student Study Habits and Academic Performance [0.0]
  The research focuses on how AI tools can support personalized learning, adaptive test adjustments, and provide real-time classroom analysis.<n>Student feedback revealed strong support for these features, and the study found a significant reduction in study hours alongside an increase in GPA.<n>Despite these benefits, challenges such as over-reliance on AI and difficulties in integrating AI with traditional teaching methods were also identified.
  arXiv  Detail & Related papers  (2024-12-03T04:51:57Z)
• Let people fail! Exploring the influence of explainable virtual and robotic agents in learning-by-doing tasks [45.23431596135002]
  This study compares the effects of classic vs. partner-aware explanations on human behavior and performance during a learning-by-doing task. Results indicated that partner-aware explanations influenced participants differently based on the type of artificial agents involved.
  arXiv  Detail & Related papers  (2024-11-15T13:22:04Z)
• Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study [36.48421439947282]
  This study designed and implemented a prompt engineering intervention at a university in Hong Kong. It examined students' AI self-efficacy, AI knowledge, and proficiency in creating effective prompts.
  arXiv  Detail & Related papers  (2024-07-30T15:05:24Z)
• Toward enriched Cognitive Learning with XAI [44.99833362998488]
  We introduce an intelligent system (CL-XAI) for Cognitive Learning which is supported by artificial intelligence (AI) tools. The use of CL-XAI is illustrated with a game-inspired virtual use case where learners tackle problems to enhance problem-solving skills.
  arXiv  Detail & Related papers  (2023-12-19T16:13:47Z)
• Towards Goal-oriented Intelligent Tutoring Systems in Online Education [69.06930979754627]
  We propose a new task, named Goal-oriented Intelligent Tutoring Systems (GITS) GITS aims to enable the student's mastery of a designated concept by strategically planning a customized sequence of exercises and assessment. We propose a novel graph-based reinforcement learning framework, named Planning-Assessment-Interaction (PAI)
  arXiv  Detail & Related papers  (2023-12-03T12:37:16Z)
• Empowering Private Tutoring by Chaining Large Language Models [87.76985829144834]
  This work explores the development of a full-fledged intelligent tutoring system powered by state-of-the-art large language models (LLMs) The system is into three inter-connected core processes-interaction, reflection, and reaction. Each process is implemented by chaining LLM-powered tools along with dynamically updated memory modules.
  arXiv  Detail & Related papers  (2023-09-15T02:42:03Z)
• Learning to Prompt in the Classroom to Understand AI Limits: A pilot study [35.06607166918901]
  Large Language Models (LLM) and the derived chatbots, like ChatGPT, have highly improved the natural language processing capabilities of AI systems. However, excitement has led to negative sentiments, even as AI methods demonstrate remarkable contributions. A pilot educational intervention was performed in a high school with 21 students.
  arXiv  Detail & Related papers  (2023-07-04T07:51:37Z)
• Reinforcement Learning Tutor Better Supported Lower Performers in a Math Task [32.6507926764587]
  Reinforcement learning could be a key tool to reduce the development cost and improve the effectiveness of intelligent tutoring software. We show that deep reinforcement learning can be used to provide adaptive pedagogical support to students learning about the concept of volume.
  arXiv  Detail & Related papers  (2023-04-11T02:11:24Z)
• Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation [43.903582264697974]
  This paper studies the problem of how to automatically, adaptively, and dynamically learn instance-level self-supervised learning strategies for each node. We propose a novel multi-teacher knowledge distillation framework for Automated Graph Self-Supervised Learning (AGSSL) Experiments on eight datasets show that AGSSL can benefit from multiple pretext tasks, outperforming the corresponding individual tasks.
  arXiv  Detail & Related papers  (2022-10-05T08:39:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.