To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks

• URL: http://arxiv.org/abs/2606.30549v1
• Date: Mon, 29 Jun 2026 16:47:46 GMT
• Title: To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks
• Abstract summary: We present Clover, a code completion tool that logs students' interactions with code suggestions.<n>Clover also offers attention checks to probe reflective engagement during programming tasks.
• Score: 1.167776800896548
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: AI code completion tools, such as Github Copilot, provide students with code suggestions to help them write programs. However, recent qualitative studies suggest that students fail to critically evaluate these suggestions. We present Clover, a code completion tool that logs students' interactions with code suggestions and additionally offers attention checks to probe reflective engagement during programming tasks. We also develop a taxonomy of behavioral interaction metrics for AI-assisted programming, informed by literature. We analyzed relationships between interaction patterns, engagement with attention checks, and task performance. We observed that higher rates of tab accept were associated with lower attention check performance, while increased dwell time was associated with higher attention check performance. We conclude by discussing how programming process data and attention checks might support reflective engagement in AI-assisted programming.

Related papers

• AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective [11.163797770670683]
  This study analyzed AI-assisted help-seeking trajectories in university-level programming.<n>Results indicate that many students primarily used AI for reactive troubleshooting rather than for planned, self-regulated problem-solving.
  arXiv  Detail & Related papers  (2026-06-22T03:37:58Z)
• Scaffolding Metacognition in Programming Education: Understanding Student-AI Interactions and Design Implications [10.81558535566768]
  Generative AI tools such as ChatGPT now provide novice programmers with unprecedented access to instant, personalized support.<n>While this holds clear promise, their influence on students' metacognitive processes remains underexplored.<n>This study addresses that gap by analyzing student-AI interactions through a metacognitive lens in university-level programming courses.
  arXiv  Detail & Related papers  (2025-11-06T07:42:24Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [60.04362496037186]
  We present the first controlled study of developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants.<n>Our results show agents can assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• Students' Reliance on AI in Higher Education: Identifying Contributing Factors [2.749898166276854]
  This study investigates potential factors contributing to patterns of AI reliance among undergraduate students.<n>appropriate reliance is significantly related to students' programming self-efficacy, programming literacy, and need for cognition.<n>Overreliance showed significant correlations with post-task trust and satisfaction with the AI assistant.
  arXiv  Detail & Related papers  (2025-06-16T17:55:26Z)
• From Coders to Critics: Empowering Students through Peer Assessment in the Age of AI Copilots [3.3094795918443634]
  This paper presents an empirical study of a rubric based, anonymized peer review process implemented in a large programming course.<n>Students evaluated each other's final projects (2D game) and their assessments were compared to instructor grades using correlation, mean absolute error, and root mean square error (RMSE)<n>Results show that peer review can approximate instructor evaluation with moderate accuracy and foster student engagement, evaluative thinking, and interest in providing good feedback to their peers.
  arXiv  Detail & Related papers  (2025-05-28T08:17:05Z)
• Improving Question Embeddings with Cognitive Representation Optimization for Knowledge Tracing [77.14348157016518]
  Research on KT modeling focuses on predicting future student performance based on existing, unupdated records of student learning interactions.<n>We propose a knowledge-tracking cognitive representation optimization (CRO-KT) model that uses dynamic programming algorithms to optimize the structure of cognitive representation.
  arXiv  Detail & Related papers  (2025-04-05T09:32:03Z)
• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.<n>Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes.<n>We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• How far are AI-powered programming assistants from meeting developers' needs? [17.77734978425295]
  In-IDE AI coding assistant tools (ACATs) like GitHub Copilot have significantly impacted developers' coding habits. We simulate real development scenarios and recruit 27 computer science students to investigate their behavior with three popular ACATs. We find that ACATs generally enhance task completion rates, reduce time, improve code quality, and increase self-perceived productivity.
  arXiv  Detail & Related papers  (2024-04-18T08:51:14Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• Composite Learning for Robust and Effective Dense Predictions [81.2055761433725]
  Multi-task learning promises better model generalization on a target task by jointly optimizing it with an auxiliary task. We find that jointly training a dense prediction (target) task with a self-supervised (auxiliary) task can consistently improve the performance of the target task, while eliminating the need for labeling auxiliary tasks.
  arXiv  Detail & Related papers  (2022-10-13T17:59:16Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)
• Enforcing Consistency in Weakly Supervised Semantic Parsing [68.2211621631765]
  We explore the use of consistency between the output programs for related inputs to reduce the impact of spurious programs. We find that a more consistent formalism leads to improved model performance even without consistency-based training.
  arXiv  Detail & Related papers  (2021-07-13T03:48:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.