AI literacy over tool design: a mixed-methods study of scaffolded versus unrestricted generative AI in programming education

• URL: http://arxiv.org/abs/2609.16784v1
• Date: Tue, 15 Sep 2026 07:53:58 GMT
• Title: AI literacy over tool design: a mixed-methods study of scaffolded versus unrestricted generative AI in programming education
• Abstract summary: This paper reports a seven-week mixed-methods pilot study in a master's-level data analytics course.<n>33 students were randomly assigned either to a scaffolded AI Study Coach embedded in the notebook-based laboratory sessions or to unrestricted use of AI tools of their own choosing.<n>The design of the tool mattered less than the students' capacity to govern their own use of it, a capacity that is at present acquired by chance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative AI has become a routine resource in programming education, and most institutional responses to it are attempts at control, either by restricting access or by offering students a controlled version of the technology. This paper reports a seven-week mixed-methods pilot study in a master's-level data analytics course, in which 33 students were randomly assigned either to a scaffolded AI Study Coach embedded in the notebook-based laboratory sessions or to unrestricted use of AI tools of their own choosing. The Coach offered stepwise hints, did not generate code, limited the number of hints per session, and required a short reflection at the end of each session. The design assumed, in line with scaffolding theory and recent experimental evidence, that guided and limited support would build confidence and reduce over-reliance, and that the scaffolded group would learn more. Assignment performance did not differ between the conditions. Students in the Coach condition reported higher confidence but managed the hint budget poorly, while students in the unrestricted condition were satisfied with their tools and uneasy about how much they depended on them. In interviews, students in both conditions identified awareness of their own reliance on AI as the most valuable outcome of the course. Students who had formulated their own rules for when to use AI performed better in both conditions, and those with the best understanding of how the models work, in every case self-taught, used the tools most deliberately and achieved the highest scores. The design of the tool mattered less than the students' capacity to govern their own use of it, a capacity that is at present acquired by chance. The paper argues that the appropriate response is structural: assessment that grades the reasoning behind AI-assisted work, and AI literacy taught explicitly as a core skill.

Related papers

• Socrates went Nuclear: Comparing Interaction Strategies for AI systems in a Learning Context using Brain Sensing [66.03354630509942]
  This paper reports on a study where we compare three designs for user-AI interaction in a learning context.<n>Fifty study participants were tasked with learning about nuclear safety protocols.
  arXiv  Detail & Related papers  (2026-09-01T02:24:35Z)
• Mitigating AI Risks in Computing Education via LLM-Driven Lecture Video Curation [42.89674478896965]
  This study evaluates the effectiveness of utilising large language models (LLMs) to retrieve targeted segments from delivered video recordings to answer student questions in programming environments.<n>We benchmarked three distinct models, two proprietary (Gemini 3.1 Pro and GPT 5.4 Pro) and one open-weight (Qwen3.5 397B), against a human lecturer's manual video selections.<n>While the AI-retrieved timestamps rarely shared exact overlaps with the human baseline, the proprietary models achieved near-parity with the expert in delivering sufficient and highly relevant answers.
  arXiv  Detail & Related papers  (2026-08-17T05:36:52Z)
• Verification-Conditioned Use: A Qualitative Study on How Generative AI Reshapes Learning, Autonomy, and Market Entry for Junior Software Developers [0.3342165620612816]
  This study investigates how the use of generative Artificial Intelligence (AI) tools affects the early stages of a career in software development.<n> sixteen themes emerged, organized around a central concept: verification-conditioned use.<n>Two themes expose tensions in newcomers' self-perception: the autonomy paradox and the first-person denial of dependence.
  arXiv  Detail & Related papers  (2026-07-27T16:07:21Z)
• "Why Put in This Much Effort?": How AI Availability Shapes Students' Motivation in Introductory Programming [0.15293427903448018]
  We study how engineering students experience and respond to AI availability.<n>Our findings complicate the assumption that students need external constraints to protect their learning.<n>Students who managed the tension found motivation in the learning process itself.
  arXiv  Detail & Related papers  (2026-06-29T15:41:43Z)
• Regulating the AI Tutor: Intentions, Help-Seeking, and Self-Regulated Learning in Adolescent GenAI Use [2.024558551027881]
  This work-in-progress examines open-ended conversational data from 98 Grade-9 students across three German Gymnasium schools.<n>We collected pre-post domain knowledge, pre-chat learning needs, and self-reported cognitive load.<n>Although students overwhelmingly selected scaffolded support before the chat, their interactions were dominated by instrumental requests.
  arXiv  Detail & Related papers  (2026-06-07T10:47:52Z)
• Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AI [47.027290803102666]
  We propose CoRe-3 (Co-Reasoning), a competency model factoring productive AI use into three assessable skills.<n>We ground the skills in theory, state five testable propositions, and instantiate them in CoReasoningLab.<n>Human-rater agreement and outcomes are next; we release the instrument, data, and protocol.
  arXiv  Detail & Related papers  (2026-06-04T10:25:31Z)
• Characterizing initial human-AI proof formalization workflows [66.06866890889131]
  Advances in AI systems' ability to generate code and engage in increasingly high-level mathematical reasoning promise to transform people's ability to formalize and thereby verify proofs.<n>We conduct a mixed-methods analysis into the initial impact of AI on people's formalization.<n>A qualitative survey shows that people's preferences are diverse, but with a general desire for AI assistance in formalization.
  arXiv  Detail & Related papers  (2026-06-02T22:58:19Z)
• Teaching Students to Question the Machine: An AI Literacy Intervention Improves Students' Regulation of LLM Use in a Science Task [0.0]
  The rapid adoption of generative artificial intelligence in schools raises concerns about students' uncritical reliance on its outputs.<n>These skills are still developing in middle school, making students vulnerable to over-trust and premature acceptance of AI outputs.<n>We report a controlled classroom study examining whether a two-hour AI literacy workshop improves students' interaction strategies and quality of final answers.
  arXiv  Detail & Related papers  (2026-04-02T12:19:43Z)
• Ensuring Computer Science Learning in the AI Era: Open Generative AI Policies and Assignment-Driven Written Quizzes [0.0]
  This paper presents an assessment model that permits the use of generative AI for take-home programming assignments.<n>To promote authentic learning, in-class, closed-book assessments are weighted more heavily than the assignments themselves.<n> Statistical analyses revealed no meaningful linear correlation between GenAI usage levels and assessment outcomes.
  arXiv  Detail & Related papers  (2026-01-16T17:02:44Z)
• "Can you feel the vibes?": An exploration of novice programmer engagement with vibe coding [42.82674998306379]
  "vibe coding" refers to creating software via natural language prompts rather than direct code authorship.<n>This paper reports on a one-day educational hackathon investigating how novice programmers and mixed-experience teams engage with vibe coding.
  arXiv  Detail & Related papers  (2025-12-02T13:32:23Z)
• Learning to Prompt in the Classroom to Understand AI Limits: A pilot study [35.06607166918901]
  Large Language Models (LLM) and the derived chatbots, like ChatGPT, have highly improved the natural language processing capabilities of AI systems. However, excitement has led to negative sentiments, even as AI methods demonstrate remarkable contributions. A pilot educational intervention was performed in a high school with 21 students.
  arXiv  Detail & Related papers  (2023-07-04T07:51:37Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.