Vibe Coding: Practice, Performance, Productivity, and Risk -A State-of-the-Art Review
- URL: http://arxiv.org/abs/2608.20446v1
- Date: Thu, 20 Aug 2026 16:10:28 GMT
- Title: Vibe Coding: Practice, Performance, Productivity, and Risk -A State-of-the-Art Review
- Abstract summary: State-of-the-art review assembles that evidence across a cross-disciplinary corpus spanning software engineering, human-computer interaction, labour economics, security research, governance, and education.<n>We survey the model landscape, the tool ecosystem, and the performance record by task type, finding the early benchmarks saturated but task-level capability uneven.<n>We document security failures in deployed applications, code-quality degradation visible in large-scale code and developer telemetry, unsettled copyright exposure, and evidence of skill atrophy.
- Score: 5.061021429555365
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Vibe coding - AI-assisted software development in which the developer describes intent in natural language and validates results by running rather than reading the generated code - was named by Andrej Karpathy in February 2025 and produced its first body of empirical evidence within seventeen months. This state-of-the-art review assembles that evidence across a cross-disciplinary corpus spanning software engineering, human-computer interaction, labour economics, security research, governance, and education. We survey the model landscape, the tool ecosystem, and the performance record by task type, finding the early benchmarks saturated but task-level capability uneven: reliable code generation alongside weak fault detection and hard-to-audit documentation. The productivity record is at first contradictory: peer-reviewed field experiments report +26% more tasks per week, independent randomised trials measure a 19% slowdown, and team-level telemetry shows code-review time up +441%. We argue these readings are consistent once measurement method, scope, and time horizon are held constant, and identify six patterns behind the dispersion, among them effect-shrinkage under broader measurement, self-report diverging from independent measurement, output volume conflated with productivity, and bold claims walked back once tested over longer horizons. We further document security failures in deployed applications, code-quality degradation visible in large-scale code and developer telemetry, unsettled copyright exposure, and evidence of skill atrophy. The review closes with the open research questions and one falsifiable conjecture: that the gains are real on new code and shrink or reverse on mature codebases, which would account for most of the disagreement in the record.
Related papers
- From Matching Models to Recruiting Agents: A Systematized Narrative Review of AI Recruitment Systems, Evaluation, and Governance [1.436449142247059]
This review traces development from bilateral retrieval and behavioral ranking through neural personjob matching.<n>We analyze three coupled transitions: from similarity to reciprocal suitability, from a model to a compound workflow, and from offline prediction to evidence- and productivity-aligned evaluation.<n>We introduce a staged mapping from evaluation evidence to the strongest defensible claim, together with an agenda for reciprocal, evidence-grounded, temporally controlled, selective, and auditable systems.
arXiv Detail & Related papers (2026-09-03T08:46:03Z) - Vibe Coding in Software Development: A Multivocal Literature Review [2.1794203097646125]
Vibe coding is a software development practice in which developers state intent in natural language and large language models generate code.<n>To our knowledge, existing reviews have not yet integrated peer-reviewed and grey literature on vibe coding.<n>This is one of the first reviews to integrate peer-reviewed and grey literature on vibe coding under a single documented protocol.
arXiv Detail & Related papers (2026-07-22T17:43:41Z) - Latent Programming Horizons in Coding Agents [6.8463578914435645]
coding agent solving a software-engineering task spends dozens of steps reasoning, editing code, and running tests.<n>We show that the residual streams of language models under coding agents linearly encode properties of the evolving program.<n> Probes trained to predict the outcome of future edits achieve performance above chance up to roughly 25 steps in advance.
arXiv Detail & Related papers (2026-07-06T15:08:26Z) - AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
arXiv Detail & Related papers (2026-05-18T17:08:26Z) - Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution [5.10403054516716]
Existing datasets evaluate performance on isolated, single pull request (PR) tasks in a stateless manner.<n>We introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS.
arXiv Detail & Related papers (2026-04-03T13:44:40Z) - Taming Imperfect Process Verifiers: A Sampling Perspective on Backtracking [54.43083499412643]
Test-time algorithms that combine the generative power of language models with process verifiers offer a promising lever for eliciting new reasoning capabilities.<n>We introduce a new process-guided test-time sampling algorithm, VGB, which uses theoretically grounded backtracking to achieve provably better robustness to verifier errors.
arXiv Detail & Related papers (2025-10-03T16:21:14Z) - CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
arXiv Detail & Related papers (2025-08-28T06:03:11Z) - DeputyDev -- AI Powered Developer Assistant: Breaking the Code Review Logjam through Contextual AI to Boost Developer Productivity [38.585498338645856]
This study investigates the implementation and efficacy of DeputyDev.<n>DeputyDev is an AI-powered code review assistant developed to address inefficiencies in the software development process.
arXiv Detail & Related papers (2025-08-13T10:09:45Z) - Training Language Models to Generate Quality Code with Program Analysis Feedback [66.0854002147103]
Code generation with large language models (LLMs) is increasingly adopted in production but fails to ensure code quality.<n>We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code.
arXiv Detail & Related papers (2025-05-28T17:57:47Z) - Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework.
Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
arXiv Detail & Related papers (2024-10-02T09:11:10Z) - Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code.
We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
arXiv Detail & Related papers (2023-02-14T18:43:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.