Evaluating the AI-Lab Intervention: Impact on Student Perception and Use of Generative AI in Early Undergraduate Computer Science Courses

• URL: http://arxiv.org/abs/2505.00100v1
• Date: Wed, 30 Apr 2025 18:12:42 GMT
• Title: Evaluating the AI-Lab Intervention: Impact on Student Perception and Use of Generative AI in Early Undergraduate Computer Science Courses
• Abstract summary: Generative AI (GenAI) is rapidly entering computer science education.<n>Concerns about overreliance coexist with a gap in research on structured scaffolding to guide tool use in formal courses.<n>This study examines the impact of a dedicated "AI-Lab" intervention on undergraduate students.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative AI (GenAI) is rapidly entering computer science education, yet its effects on student learning, skill development, and perceptions remain underexplored. Concerns about overreliance coexist with a gap in research on structured scaffolding to guide tool use in formal courses. This study examines the impact of a dedicated "AI-Lab" intervention -- emphasizing guided scaffolding and mindful engagement -- on undergraduate students in Data Structures and Algorithms, Competitive Programming, and first-year engineering courses at Purdue University. Over three semesters, we integrated AI-Lab modules into four mandatory and elective courses, yielding 831 matched pre- and post-intervention survey responses, alongside focus group discussions. Employing a mixed-methods approach, we analyzed quantitative shifts in usage patterns and attitudes as well as qualitative narratives of student experiences. While the overall frequency of GenAI usage for homework or programming projects remained largely stable, we observed large effect sizes in comfort and openness across conceptual, debugging, and homework problems. Notably, usage patterns for debugging also shifted statistically significantly, reflecting students' more mindful and deliberate approach. Focus group discussions corroborated these results, suggesting that the intervention "bridged the gap" between naive GenAI usage and more nuanced, reflective integration of AI tools into coursework, ultimately heightening students' awareness of their own skill development. These findings suggest that structured, scaffolded interventions can enable students to harness GenAI's benefits without undermining essential competencies. We offer evidence-based recommendations for educators seeking to integrate GenAI responsibly into computing curricula and identify avenues for future research on GenAI-supported pedagogy.

Related papers

• Quick Build, Careful Check? Generative AI Use in Hackathons [83.0493885475068]
  We report an exploratory interview study with participants at an AI-themed hackathon in central Europe.<n>We found that participants used GenAI for purposes beyond coding, including learning unfamiliar topics, brainstorming, and preparing documentation.<n>We also found that all participants converged on an unwritten practice of checking GenAI output before using it, yet their ability to actually verify was constrained by time pressure and limited domain knowledge.
  arXiv  Detail & Related papers  (2026-07-31T08:59:18Z)
• Is Solving Better Than Evaluating GenAI Solutions? [45.714140455691485]
  This study compares evaluating GenAI-generated solutions with traditional problem solving in a junior-level algorithms course.<n>We found no statistically significant differences between groups in midterm scores, final exam scores, overall course grades, or exam problems structurally aligned with the homework interventions.<n>Students received significantly higher homework scores when evaluating GenAI-generated solutions, but this localized advantage did not translate into downstream gains.
  arXiv  Detail & Related papers  (2026-07-30T02:08:29Z)
• Patterns of Learner-AI Interaction and Academic Performance in an Object-Oriented Programming Course [0.0]
  This research paper examines how different forms of learner-AI interaction relate to learning outcomes in object-oriented programming (OOP) courses.<n>Survey data were collected from 210 undergraduate students enrolled in a first-year OOP course.
  arXiv  Detail & Related papers  (2026-05-26T07:48:00Z)
• How Do Software Engineering Students Use Generative AI in Real-World Capstone Projects? An Empirical Baseline Study [1.5102168012018966]
  The rapid integration of generative AI (GenAI) into professional software development adds new challenges.<n>We conducted a large-scale study of self-determined GenAI use in an undergraduate RWCP course.<n>Our findings provide a characterization of GenAI practices across the software engineering lifecycle.
  arXiv  Detail & Related papers  (2026-04-27T14:23:46Z)
• Exploring Student Perception on Gen AI Adoption in Higher Education: A Descriptive Study [1.978884131103313]
  This study examines how students perceive, use, and evaluate Generative Artificial Intelligence (GenAI)<n>Data were collected through a questionnaire administered to 436 postgraduate Computer Science students at the University of Hertfordshire.
  arXiv  Detail & Related papers  (2026-03-29T17:19:39Z)
• AI Combines, Humans Socialise: A SECI-based Experience Report on Business Simulation Games [0.0]
  This paper reports on the integration of generative AI tools into a Business Simulation Games (BSG) designed for engineering students.<n> AI was embedded as a support mechanism during the simulation to assist students in analysing events, reformulating information, and generating decision-relevant insights.<n>The results suggest a functional boundary in human-AI collaboration within simulation-based learning.
  arXiv  Detail & Related papers  (2026-02-24T07:26:16Z)
• AI Literacy, Safety Awareness, and STEM Career Aspirations of Australian Secondary Students: Evaluating the Impact of Workshop Interventions [38.350232667249095]
  Deepfakes and other forms of synthetic media pose growing safety risks for adolescents.<n>This study evaluates the impact of Day of AI Australia's workshop-based intervention on Australian secondary students.
  arXiv  Detail & Related papers  (2026-01-30T02:55:53Z)
• Ensuring Computer Science Learning in the AI Era: Open Generative AI Policies and Assignment-Driven Written Quizzes [0.0]
  This paper presents an assessment model that permits the use of generative AI for take-home programming assignments.<n>To promote authentic learning, in-class, closed-book assessments are weighted more heavily than the assignments themselves.<n> Statistical analyses revealed no meaningful linear correlation between GenAI usage levels and assessment outcomes.
  arXiv  Detail & Related papers  (2026-01-16T17:02:44Z)
• Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming [3.2383459424229835]
  This study investigates how GenAI assistance compares to conventional online resources in supporting knowledge gains across different proficiency levels.<n>We conducted a controlled user experiment with 24 undergraduate students of two different levels of programming experience.<n>Our findings reveal that generating complete solutions with GenAI significantly improves task performance, especially for beginners, but does not consistently result in knowledge gains.
  arXiv  Detail & Related papers  (2025-11-17T11:42:24Z)
• Short-Term Gains, Long-Term Gaps: The Impact of GenAI and Search Technologies on Retention [1.534667887016089]
  This study investigates how GenAI (ChatGPT), search engines (Google), and e-textbooks influence student performance across tasks of varying cognitive complexity.<n>ChatGPT and Google groups outperformed the control group in immediate assessments for lower-order cognitive tasks.<n>While AI-driven tools facilitate immediate performance, they do not inherently reinforce long-term retention unless supported by structured learning strategies.
  arXiv  Detail & Related papers  (2025-07-10T00:44:50Z)
• Integrating Universal Generative AI Platforms in Educational Labs to Foster Critical Thinking and Digital Literacy [0.3749861135832073]
  This paper presents a new educational framework for integrating generative artificial intelligence (GenAI) platforms into laboratory activities.<n> Recognizing the limitations and risks of uncritical reliance on large language models (LLMs), the proposed pedagogical model reframes GenAI as a research subject and cognitive tool.
  arXiv  Detail & Related papers  (2025-06-11T17:45:51Z)
• Encouraging Students' Responsible Use of GenAI in Software Engineering Education: A Causal Model and Two Institutional Applications [1.1511012020557325]
  generative AI (GenAI) tools such as ChatGPT and GitHub Copilot become pervasive in education.<n>Concerns are rising about students using them to complete rather than learn from coursework.<n>This paper proposes and empirically applies a causal model to help educators scaffold responsible GenAI use in Software Engineering education.
  arXiv  Detail & Related papers  (2025-05-31T19:27:40Z)
• From Recall to Reasoning: Automated Question Generation for Deeper Math Learning through Large Language Models [44.99833362998488]
  We investigated the first steps for optimizing content creation for advanced math.<n>We looked at the ability of GenAI to produce high-quality practice problems that are relevant to the course content.
  arXiv  Detail & Related papers  (2025-05-17T08:30:10Z)
• Methodological Foundations for AI-Driven Survey Question Generation [41.94295877935867]
  This paper presents a methodological framework for using generative AI in educational survey research.<n>We explore how Large Language Models can generate adaptive, context-aware survey questions.<n>We examine ethical issues such as bias, privacy, and transparency.
  arXiv  Detail & Related papers  (2025-05-02T09:50:34Z)
• Student's Use of Generative AI as a Support Tool in an Advanced Web Development Course [0.5371337604556311]
  We analyze the use of GenAI as a support tool for learning, creativity, and productivity in a web development course for undergraduate students.<n>Students used GenAI on different tasks with a reported increase in learning and productivity.
  arXiv  Detail & Related papers  (2025-03-19T20:34:21Z)
• Computational Safety for Generative AI: A Signal Processing Perspective [65.268245109828]
  computational safety is a mathematical framework that enables the quantitative assessment, formulation, and study of safety challenges in GenAI.<n>We show how sensitivity analysis and loss landscape analysis can be used to detect malicious prompts with jailbreak attempts.<n>We discuss key open research challenges, opportunities, and the essential role of signal processing in computational AI safety.
  arXiv  Detail & Related papers  (2025-02-18T02:26:50Z)
• Navigating Ethical Challenges in Generative AI-Enhanced Research: The ETHICAL Framework for Responsible Generative AI Use [0.0]
  The rapid adoption of generative artificial intelligence (GenAI) in research presents both opportunities and ethical challenges.<n>This paper develops the ETHICAL framework, which is a practical guide for responsible GenAI use in research.
  arXiv  Detail & Related papers  (2024-12-11T05:49:11Z)
• Early Adoption of Generative Artificial Intelligence in Computing Education: Emergent Student Use Cases and Perspectives in 2023 [38.83649319653387]
  There is limited prior research on computing students' use and perceptions of GenAI. We surveyed all computer science majors in a small engineering-focused R1 university. We discuss the impact of our findings on the emerging conversation around GenAI and education.
  arXiv  Detail & Related papers  (2024-11-17T20:17:47Z)
• LLMs Integration in Software Engineering Team Projects: Roles, Impact, and a Pedagogical Design Space for AI Tools in Computing Education [7.058964784190549]
  This work takes a pedagogical lens to explore the implications of generative AI (GenAI) models and tools, such as ChatGPT and GitHub Copilot. Our results address a particular gap in understanding the role and implications of GenAI on teamwork, team-efficacy, and team dynamics.
  arXiv  Detail & Related papers  (2024-10-30T14:43:33Z)
• Hey GPT, Can You be More Racist? Analysis from Crowdsourced Attempts to Elicit Biased Content from Generative AI [41.96102438774773]
  This work presents the findings from a university-level competition, which challenged participants to design prompts for eliciting biased outputs from GenAI tools. We quantitatively and qualitatively analyze the competition submissions and identify a diverse set of biases in GenAI and strategies employed by participants to induce bias in GenAI.
  arXiv  Detail & Related papers  (2024-10-20T18:44:45Z)
• Enhancing Feature Selection and Interpretability in AI Regression Tasks Through Feature Attribution [38.53065398127086]
  This study investigates the potential of feature attribution methods to filter out uninformative features in input data for regression problems. We introduce a feature selection pipeline that combines Integrated Gradients with k-means clustering to select an optimal set of variables from the initial data space. To validate the effectiveness of this approach, we apply it to a real-world industrial problem - blade vibration analysis in the development process of turbo machinery.
  arXiv  Detail & Related papers  (2024-09-25T09:50:51Z)
• BoilerTAI: A Platform for Enhancing Instruction Using Generative AI in Educational Forums [0.0]
  This paper describes a practical, scalable platform that seamlessly integrates Generative AI (GenAI) with online educational forums. The platform empowers instructional staff to efficiently manage, refine, and approve responses by facilitating interaction between student posts and a Large Language Model (LLM)
  arXiv  Detail & Related papers  (2024-09-20T04:00:30Z)
• Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study [36.48421439947282]
  This study designed and implemented a prompt engineering intervention at a university in Hong Kong. It examined students' AI self-efficacy, AI knowledge, and proficiency in creating effective prompts.
  arXiv  Detail & Related papers  (2024-07-30T15:05:24Z)
• Model-based Maintenance and Evolution with GenAI: A Look into the Future [47.93555901495955]
  We argue that Generative Artificial Intelligence (GenAI) can be used as a means to address the limitations of Model-Based Engineering (MBM&E) We propose that GenAI can be used in MBM&E for: reducing engineers' learning curve, maximizing efficiency with recommendations, or serving as a reasoning tool to understand domain problems.
  arXiv  Detail & Related papers  (2024-07-09T23:13:26Z)
• OpenHEXAI: An Open-Source Framework for Human-Centered Evaluation of Explainable Machine Learning [43.87507227859493]
  This paper presents OpenHEXAI, an open-source framework for human-centered evaluation of XAI methods. OpenHEAXI is the first large-scale infrastructural effort to facilitate human-centered benchmarks of XAI methods.
  arXiv  Detail & Related papers  (2024-02-20T22:17:59Z)
• Toward enriched Cognitive Learning with XAI [44.99833362998488]
  We introduce an intelligent system (CL-XAI) for Cognitive Learning which is supported by artificial intelligence (AI) tools. The use of CL-XAI is illustrated with a game-inspired virtual use case where learners tackle problems to enhance problem-solving skills.
  arXiv  Detail & Related papers  (2023-12-19T16:13:47Z)
• Innovating Computer Programming Pedagogy: The AI-Lab Framework for Generative AI Adoption [0.0]
  We introduce "AI-Lab," a framework for guiding students in effectively leveraging GenAI within core programming courses. By identifying and rectifying GenAI's errors, students enrich their learning process. For educators, AI-Lab provides mechanisms to explore students' perceptions of GenAI's role in their learning experience.
  arXiv  Detail & Related papers  (2023-08-23T17:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.