Vibe Coding: An Experiment with Test-Driven Development

• URL: http://arxiv.org/abs/2607.22406v1
• Date: Fri, 24 Jul 2026 15:24:57 GMT
• Title: Vibe Coding: An Experiment with Test-Driven Development
• Abstract summary: Conversational Large Language Models (CLLMs) can automatically generate code by collaborating with users through natural language.<n>Poor collaboration can lead to poor quality output.<n>This study aims to investigate how humans and CLLMs can collaborate as peers through vibe coding.
• Score: 0.4804905638984918
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Context: Conversational Large Language Models (CLLMs) can automatically generate code by collaborating with users through natural language. However, poor collaboration can lead to poor quality output. Objective: This exploratory study aims to investigate how humans and CLLMs can collaborate as peers through vibe coding, an approach that integrates principles from prompt engineering, agile design, and human-AI co-creation to enhance collaboration. Method: We designed four interaction models representing different collaboration patterns in the software development process: the solo model (human-only development), the collaborative model (human-CLLM collaboration), the fully automated model (development autonomously performed by a CLLM), and the agentic model (development autonomously performed by the MetaGPT~X platform). Based on these models, we implemented corresponding Test-Driven Development (TDD) workflows using structured prompts and Python scripts. We then conducted a controlled pre-experimental study with TDD professionals to compare the solo and collaborative workflows. In addition, we performed repeated exploratory executions of fully automated and agentic workflows on the same development tasks to obtain complementary evidence. Results: Our findings suggest that the choice of interaction model should depend on the development objective. Agentic workflows are best suited for rapid development and functionally correct production code but may introduce additional implementation complexity. However, they may also introduce additional implementation decisions that are not explicitly required by the functional specifications, resulting in untested decision points. In contrast, collaborative workflows produce higher-quality, better-organized test suites. Conclusions: Our work explored how...

Related papers

• Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration [54.18136393879849]
  We present ALMANAC, a dataset of action-level mental model ANnotations for Agent Collaboration.<n>ALMANAC contains 2,987 collaboration actions, each paired with theory-informed mental model annotations.<n>Our results demonstrate ALMANAC's utility in evaluating models' ability to simulate human collaborative behaviors.
  arXiv  Detail & Related papers  (2026-06-04T16:56:12Z)
• Pragmos: A Process Agentic Modeling System [0.0]
  Large Language Models (LLMs) are now being explored as tools to derive process models directly from textual descriptions.<n>We envision modeling as an open-ended conversational activity, best supported by an interactive, iterative process involving both humans and LLM.<n>We present our research agenda and introduce Pragmos, a prototype system that operationalizes this vision.
  arXiv  Detail & Related papers  (2026-04-30T01:49:19Z)
• EmCoop: A Framework and Benchmark for Embodied Cooperation Among LLM Agents [18.802912315746564]
  EmCoop is a benchmark framework for studying cooperation in embodied multi-agent systems.<n>Our framework separates a high-level cognitive layer from a low-level embodied interaction layer.<n>We propose generalizable, process-level metrics that diagnose collaboration quality and failure modes.
  arXiv  Detail & Related papers  (2026-02-27T22:28:33Z)
• MoCo: A One-Stop Shop for Model Collaboration Research [132.52160996841505]
  We present MoCo: a one-stop Python library of executing, benchmarking, and comparing model collaboration algorithms at scale.<n>MoCo features 26 model collaboration methods, spanning diverse levels of cross-model information exchange.<n>Extensive experiments with MoCo demonstrate that most collaboration strategies outperform models without collaboration.<n>We envision MoCo as a valuable toolkit to facilitate and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.
  arXiv  Detail & Related papers  (2026-01-29T04:36:52Z)
• Seismology modeling agent: A smart assistant for geophysical researchers [14.28965530601497]
  This paper proposes an intelligent, interactive workflow powered by Large Language Models (LLMs)<n>We introduce the first Model Context Protocol (MCP) server suite for SPECFEM.<n>The framework supports both fully automated execution and human-in-the-loop collaboration.
  arXiv  Detail & Related papers  (2025-12-16T14:18:26Z)
• A Survey of Vibe Coding with Large Language Models [93.88284590533242]
  "Vibe Coding" is a development methodology where developers validate AI-generated implementations through outcome observation.<n>Despite its transformative potential, the effectiveness of this emergent paradigm remains under-explored.<n>This survey provides the first comprehensive and systematic review of Vibe Coding with large language models.
  arXiv  Detail & Related papers  (2025-10-14T11:26:56Z)
• Cross-Task Experiential Learning on LLM-based Multi-Agent Collaboration [63.90193684394165]
  We introduce multi-agent cross-task experiential learning (MAEL), a novel framework that endows LLM-driven agents with explicit cross-task learning and experience accumulation.<n>During the experiential learning phase, we quantify the quality for each step in the task-solving workflow and store the resulting rewards.<n>During inference, agents retrieve high-reward, task-relevant experiences as few-shot examples to enhance the effectiveness of each reasoning step.
  arXiv  Detail & Related papers  (2025-05-29T07:24:37Z)
• Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI [0.36868085124383626]
  Review presents a comprehensive analysis of two emerging paradigms in AI-assisted software development: vibe coding and agentic coding.<n> Vibe coding emphasizes intuitive, human-in-the-loop interaction through prompt-based, conversational interaction.<n>Agentic coding enables autonomous software development through goal-driven agents capable of planning, executing, testing, and iterating tasks with minimal human intervention.
  arXiv  Detail & Related papers  (2025-05-26T03:00:21Z)
• ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems [80.69865295743149]
  This work attempts to study using LLM-based agents to design collaborative AI systems autonomously.<n>Based on ComfyBench, we develop ComfyAgent, a framework that empowers agents to autonomously design collaborative AI systems by generating.<n>While ComfyAgent achieves a comparable resolve rate to o1-preview and significantly surpasses other agents on ComfyBench, ComfyAgent has resolved only 15% of creative tasks.
  arXiv  Detail & Related papers  (2024-09-02T17:44:10Z)
• Multi-Agent Collaboration via Cross-Team Orchestration [31.506350304184526]
  Large Language Models (LLMs) have significantly impacted various domains, especially through organized autonomous agents.<n>We introduce Cross-Team Orchestration (Croto), a scalable multi-team framework that enables orchestrated teams to jointly propose various task-oriented solutions.<n>Experiments reveal a notable increase in software quality compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-06-13T10:18:36Z)
• Large Language Model-based Human-Agent Collaboration for Complex Task Solving [94.3914058341565]
  We introduce the problem of Large Language Models (LLMs)-based human-agent collaboration for complex task-solving. We propose a Reinforcement Learning-based Human-Agent Collaboration method, ReHAC. This approach includes a policy model designed to determine the most opportune stages for human intervention within the task-solving process.
  arXiv  Detail & Related papers  (2024-02-20T11:03:36Z)
• Self-collaboration Code Generation via ChatGPT [35.88318116340547]
  Large Language Models (LLMs) have demonstrated remarkable code-generation ability, but struggle with complex tasks. We present a self-collaboration framework for code generation employing LLMs, exemplified by ChatGPT. To effectively organize and manage this virtual team, we incorporate software-development methodology into the framework.
  arXiv  Detail & Related papers  (2023-04-15T16:33:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.