Vibe Coding Ate My Homework: An evaluation of AI approaches to greenfield software engineering and programming

• URL: http://arxiv.org/abs/2606.18293v1
• Date: Mon, 15 Jun 2026 18:49:02 GMT
• Title: Vibe Coding Ate My Homework: An evaluation of AI approaches to greenfield software engineering and programming
• Abstract summary: This paper aims to evaluate the viability of vibe coding for greenfield software engineering tasks.<n>We have developed an evaluation suite for analysing an LLM's proficiency in carrying out simple, isolated greenfield programming tasks in Python.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Thanks to rapid developments in generative AI, we are in the midst of a paradigm shift that may change how we interact with computers forever. We have observed a growth in the use of natural language prompts to build applications and coding infrastructures without underlying knowledge of the field, and this practice has been dubbed `vibe coding.' It arguably represents what the field of programming has been building towards since the beginning, with every higher level of abstraction that is conceived. Vibe coding promises to be the endpoint for the meta of high-level programming as far as method of input is concerned: eliminating a human's use of code syntax entirely in favour of programming in their mother tongue. This paper aims to evaluate the viability of vibe coding for greenfield software engineering tasks, as well as analyse the benchmarks that have been used to measure its software engineering prowess. To this end, we have developed an evaluation suite for analysing an LLM's proficiency in carrying out simple, isolated greenfield programming tasks in Python to provide scoped insight on the matter.

Related papers

• Beyond the Traceback: Using LLMs for Adaptive Explanations of Programming Errors [52.619793817715326]
  We present a multi-stage crowdsourced study evaluating skill-targeted, LLM-generated Python error messages.<n>We show that while LLM-rewritten messages significantly improved subjective evaluations, these perceived gains did not translate into statistically significant improvements in objective debug performance.<n>This highlights a critical human-AI complementarity gap: explanations that feel better to users do not necessarily make them more effective debuggers.
  arXiv  Detail & Related papers  (2026-08-21T09:14:28Z)
• Computer Science Achievement and Writing Skills Predict Vibe Coding Proficiency [5.449649175237595]
  "vibe coding" allows one to specify programs in natural language and iterate from observed behavior without editing source code.<n>While its adoption is accelerating, little is known about which skills best predict success in this workflow.<n>We report a preregistered cross-sectional study with tertiary-level students who completed measures of computer-science achievement, domain-general cognitive skills, written-communication proficiency, and a vibe-coding assessment.
  arXiv  Detail & Related papers  (2026-03-14T21:42:22Z)
• "Can you feel the vibes?": An exploration of novice programmer engagement with vibe coding [42.82674998306379]
  "vibe coding" refers to creating software via natural language prompts rather than direct code authorship.<n>This paper reports on a one-day educational hackathon investigating how novice programmers and mixed-experience teams engage with vibe coding.
  arXiv  Detail & Related papers  (2025-12-02T13:32:23Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• Vibe Coding: Toward an AI-Native Paradigm for Semantic and Intent-Driven Programming [0.0]
  This paper introduces vibe coding, an emerging AI-native programming paradigm in which a developer specifies high-level functional intent along with qualitative descriptors of the desired "vibe"<n>An intelligent agent then transforms those specifications into executable software.
  arXiv  Detail & Related papers  (2025-10-09T22:31:53Z)
• Generative AI for Object-Oriented Programming: Writing the Right Code and Reasoning the Right Logic [2.170361965861349]
  Large language models (LLMs) have diverse applications spanning finance, commonsense knowledge graphs, medicine, and visual analysis.<n>Our work aims to address this gap by presenting a vision from the perspectives of key stakeholders involved in an OOP task.<n> Furthermore, we propose ways to augment existing logical reasoning and code writing, ultimately enhancing the programming experience.
  arXiv  Detail & Related papers  (2025-08-07T03:38:17Z)
• MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks [56.34018316319873]
  We propose MERA Code, a benchmark for evaluating code for the latest code generation LLMs in Russian.<n>This benchmark includes 11 evaluation tasks that span 8 programming languages.<n>We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages.
  arXiv  Detail & Related papers  (2025-07-16T14:31:33Z)
• Challenges and Paths Towards AI for Software Engineering [55.95365538122656]
  We discuss progress in AI for software engineering in threefold manner.<n>First, we provide a structured taxonomy of concrete tasks in AI for software engineering.<n>Second, we outline several key bottlenecks that limit current approaches.
  arXiv  Detail & Related papers  (2025-03-28T17:17:57Z)
• Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models? [20.275891144535258]
  We explore the extent to which "traditional" programming languages remain relevant for non-expert end-user programmers in a world with generative AI. We outline some reasons that traditional programming languages may still be relevant and useful for end-user programmers.
  arXiv  Detail & Related papers  (2023-11-01T09:20:21Z)
• Towards AGI in Computer Vision: Lessons Learned from GPT and Large Language Models [98.72986679502871]
  Chat systems powered by large language models (LLMs) emerge and rapidly become a promising direction to achieve artificial general intelligence (AGI) But the path towards AGI in computer vision (CV) remains unclear. We imagine a pipeline that puts a CV algorithm in world-scale, interactable environments, pre-trains it to predict future frames with respect to its action, and then fine-tunes it with instruction to accomplish various tasks.
  arXiv  Detail & Related papers  (2023-06-14T17:15:01Z)
• Is ChatGPT the Ultimate Programming Assistant -- How far is it? [11.943927095071105]
  ChatGPT has received great attention: it can be used as a bot for discussing source code. We present an empirical study of ChatGPT's potential as a fully automated programming assistant.
  arXiv  Detail & Related papers  (2023-04-24T09:20:13Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.