Changes in Coding Behavior and Performance Since the Introduction of LLMs

• URL: http://arxiv.org/abs/2601.11835v1
• Date: Fri, 16 Jan 2026 23:51:31 GMT
• Title: Changes in Coding Behavior and Performance Since the Introduction of LLMs
• Abstract summary: We analyze five years of student source code submissions in a graduate-level cloud computing course.<n>Student coding behavior has changed significantly since Fall 2022.
• Score: 1.7482685917723753
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The widespread availability of large language models (LLMs) has changed how students engage with coding and problem-solving. While these tools may increase student productivity, they also make it more difficult for instructors to assess students' learning and effort. In this quasi-longitudinal study, we analyze five years of student source code submissions in a graduate-level cloud computing course, focusing on an assignment that remained unchanged and examining students' behavior during the period spanning five semesters before the release of ChatGPT and five semesters after. Student coding behavior has changed significantly since Fall 2022. The length of their final submissions increased. Between consecutive submissions, average edit distances increased while average score improvement decreased, suggesting that both student productivity and learning have decreased after ChatGPT's release. Additionally, there are statistically significant correlations between these behavioral changes and their overall performance. Although we cannot definitively attribute them to LLM misuse, they are consistent with our hypothesis that some students are over-reliant on LLMs, which is negatively affecting their learning outcomes. Our findings raise an alarm around the first generation of graduates in the age of LLMs, calling upon both educators and employers to reflect on their evaluation methods for genuine expertise and productivity.

Related papers

• Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors [52.619793817715326]
  We present a multi-stage crowdsourced study evaluating skill-targeted, LLM-generated Python error messages.<n>We show that while LLM-rewritten messages significantly improved subjective evaluations, these perceived gains did not translate into statistically significant improvements in objective debug performance.<n>This highlights a critical human-AI complementarity gap: explanations that feel better to users do not necessarily make them more effective debuggers.
  arXiv  Detail & Related papers  (2026-08-21T09:14:28Z)
• Less Deliberate in Teams: Student LLM Use Across Individual and Collaborative Work [0.0]
  This paper reports findings from a semester-long study of 96 undergraduate students who completed six assignments.<n>LLMs usage dropped by 42.7 percentage points when students moved from individual work to their first team milestone.<n>Students also wrote fewer and simpler prompts, used fewer intentional prompting strategies, and checked LLM output less carefully.
  arXiv  Detail & Related papers  (2026-06-29T19:45:35Z)
• The Crutch or the Ceiling? How Different Generations of LLMs Shape EFL Student Writings [0.26097841018267615]
  This study explores the extent and limitations of Large Language Models (LLMs) in assisting secondary-level English as a Foreign Language (EFL) students with their writing tasks.<n>Our results indicate that advanced LLMs boost assessment scores and lexical diversity for lower-proficiency learners.<n>To transform AI-assisted practice into genuine learning, pedagogy must shift from focusing on output quality to verifying the learning process.
  arXiv  Detail & Related papers  (2026-04-16T18:19:46Z)
• Can Large Language Models Help Students Prove Software Correctness? An Experimental Study with Dafny [75.55915044740566]
  Students in computing education increasingly use large language models (LLMs) such as ChatGPT.<n>This paper investigates how students interact with an LLM when solving formal verification exercises in Dafny.
  arXiv  Detail & Related papers  (2025-06-27T16:34:13Z)
• DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs [58.4911494598431]
  DistiLLM-2 is a contrastive approach that simultaneously increases the likelihood of teacher responses and decreases that of student responses.<n>Our experiments show that DistiLLM-2 not only builds high-performing student models across a wide range of tasks, but also supports diverse applications.
  arXiv  Detail & Related papers  (2025-03-10T08:51:32Z)
• What You See Is Not Always What You Get: An Empirical Study of Code Comprehension by Large Language Models [0.5735035463793009]
  We investigate the vulnerability of large language models (LLMs) to imperceptible attacks, where hidden character manipulation in source code misleads LLMs' behaviour while remaining undetectable to human reviewers.<n>These attacks include coding reordering, invisible coding characters, code deletions, and code homoglyphs.<n>Our findings confirm the susceptibility of LLMs to imperceptible coding character attacks, while different LLMs present different negative correlations between perturbation magnitude and performance.
  arXiv  Detail & Related papers  (2024-12-11T04:52:41Z)
• Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation [42.49889252988544]
  Large language model (LLM)-based programming assistants can help improve the productivity of professional software developers, but can also facilitate cheating in introductory computer programming courses. This paper investigates the baseline performance of 5 widely used LLMs on a collection of introductory programming problems, examines adversarial perturbations to degrade their performance, and describes the results of a user study aimed at understanding the efficacy of such perturbations in hindering actual code generation for introductory programming assignments.
  arXiv  Detail & Related papers  (2024-10-12T01:01:00Z)
• Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course [0.9718746651638346]
  Large language models (LLMs) can generate, debug, and explain code. Our study explores how students' social perceptions influence their own LLM usage.
  arXiv  Detail & Related papers  (2024-06-10T16:40:14Z)
• The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances [28.182426634762496]
  Large language models (LLMs) are quickly being adopted in a wide range of learning experiences.<n>We conducted a large-scale randomized control trial with 5,831 students from 146 countries in an online coding class.<n>We estimate positive benefits on exam performance for adopters, the students who used the tool, but over all students, the advertisement of GPT-4 led to a significant average decrease in exam participation.
  arXiv  Detail & Related papers  (2024-04-25T15:39:22Z)
• The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition [74.04775677110179]
  In-context Learning (ICL) has emerged as a powerful paradigm for performing natural language tasks with Large Language Models (LLM) We show that LLMs have strong yet inconsistent priors in emotion recognition that ossify their predictions. Our results suggest that caution is needed when using ICL with larger LLMs for affect-centered tasks outside their pre-training domain.
  arXiv  Detail & Related papers  (2024-03-25T19:07:32Z)
• GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers [68.77382332826167]
  Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations.
  arXiv  Detail & Related papers  (2024-02-29T15:26:14Z)
• Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE [62.13435256279566]
  Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks. However, their large size makes their inference slow and computationally expensive. We show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer.
  arXiv  Detail & Related papers  (2023-10-28T04:07:58Z)
• How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging [28.321080454393687]
  Large Language Models (LLMs) now excel at generative skills and can create content at impeccable speeds. Human novices play the role of Teaching Assistants and help LLM-powered teachable agents code. We introduce Hypo, a novel system to facilitate deliberate practice on debug, where human novices play the role of Teaching Assistants and help LLM-powered teachable agents code.
  arXiv  Detail & Related papers  (2023-10-08T21:39:47Z)
• Identifying Different Student Clusters in Functional Programming Assignments: From Quick Learners to Struggling Students [2.0386745041807033]
  We analyze student assignment submission data collected from a functional programming course taught at McGill university. This allows us to identify four clusters of students: "Quick-learning", "Hardworking", "Satisficing", and "Struggling" We then analyze how work habits, working duration, the range of errors, and the ability to fix errors impact different clusters of students.
  arXiv  Detail & Related papers  (2023-01-06T17:15:58Z)
• Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble [56.705249154629264]
  Self-training teacher-student frameworks are proposed to improve the robustness of NER models. In this paper, we propose an adaptive teacher learning comprised of two teacher-student networks. Fine-grained student ensemble updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise.
  arXiv  Detail & Related papers  (2022-12-13T12:14:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.