Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts

• URL: http://arxiv.org/abs/2409.12447v2
• Date: Fri, 25 Apr 2025 03:55:39 GMT
• Title: Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
• Abstract summary: We argue that some prompts are programs and that the development of prompts is a distinct phenomenon in programming known as "prompt programming"<n>We develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development.
• Score: 12.979519344826748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative pre-trained models power intelligent software features used by millions of users controlled by developer-written natural language prompts. Despite the impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some prompts are programs and that the development of prompts is a distinct phenomenon in programming known as "prompt programming". We develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt structures. We contribute 15 observations to form a preliminary understanding of current prompt programming practices. For example, rather than building mental models of code, prompt programmers develop mental models of the foundation model (FM)'s behavior on the prompt by interacting with the FM. While prior research shows that experts have well-formed mental models, we find that prompt programmers who have developed dozens of prompts still struggle to develop reliable mental models. Our observations show that prompt programming differs from traditional software development, motivating the creation of prompt programming tools and providing implications for software engineering stakeholders.

Related papers

• Imprompt: A Language Framework for Prompt Programming [0.8984888893275712]
  We propose Imprompt, a new language framework for the study and practice of prompt programming.<n>We argue that prompt programs must contain only the task descriptions and must be decoupled from lower-level 'execution' details.<n>We then explore the idea of typing for prompt programs and draw a correspondence between type checking and constrained decoding.
  arXiv  Detail & Related papers  (2026-07-13T05:02:43Z)
• REprompt: Prompt Generation for Intelligent Software Development Guided by Requirements Engineering [43.0808976544794]
  Large language models increasingly function as foundation models within coding agents.<n> prompts play a central role in agent-based intelligent software development.<n>We propose REprompt, a multi-agent prompt optimization framework guided by requirements engineering.
  arXiv  Detail & Related papers  (2026-01-23T07:14:34Z)
• Guidelines to Prompt Large Language Models for Code Generation: An Empirical Characterization [82.29178197694819]
  We derive and evaluate development-specific prompt optimization guidelines.<n>We use an iterative, test-driven approach to automatically refine code generation prompts.<n>We conduct an assessment with 50 practitioners, who report their usage of the elicited prompt improvement patterns.
  arXiv  Detail & Related papers  (2026-01-19T15:01:42Z)
• Building Software by Rolling the Dice: A Qualitative Study of Vibe Coding [15.145249560710377]
  "vibe coders" build software primarily through prompts rather than writing code.<n>We conducted a theory study of 20 vibe-coding videos, including 7 live-streamed coding sessions and 13 opinion videos.<n>Our findings reveal a spectrum of behaviors: some vibe coders rely almost entirely on AI, while others examine and adapt generated outputs.
  arXiv  Detail & Related papers  (2025-12-27T00:38:37Z)
• Understanding Prompt Programming Tasks and Questions [19.414347044995644]
  We develop a taxonomy of 25 tasks prompt programmers do and 51 questions they ask, measuring the importance of each task and question.<n>We find that prompt programming is not well-supported: all tasks are done manually, and 16 of the 51 questions -- including a majority of the most important ones -- remain unanswered.
  arXiv  Detail & Related papers  (2025-07-23T07:01:44Z)
• NL-Debugging: Exploiting Natural Language as an Intermediate Representation for Code Debugging [68.42255321759062]
  Recent advancements in large language models (LLMs) have shifted attention toward leveraging natural language reasoning to enhance code-related tasks.<n>In this paper, we introduce NL-GING, a novel framework that employs natural language as an intermediate representation to improve code.
  arXiv  Detail & Related papers  (2025-05-21T10:38:50Z)
• Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Models [22.339868419855904]
  Students increasingly rely on generative AI tools for programming assistance, often without formal instruction or guidance. This highlights a need to teach students how to effectively interact with AI models. We developed a novel platform for prompt programming that enables authentic dialogue-based interactions.
  arXiv  Detail & Related papers  (2025-03-06T09:56:07Z)
• Promptware Engineering: Software Engineering for LLM Prompt Development [22.788377588087894]
  Large Language Models (LLMs) are increasingly integrated into software applications, with prompts serving as the primary 'programming' interface. As a result, a new software paradigm, promptware, has emerged, using natural language prompts to interact with LLMs. Unlike traditional software, which relies on formal programming languages and deterministic runtime environments, promptware is based on ambiguous, unstructured, and context-dependent natural language.
  arXiv  Detail & Related papers  (2025-03-04T08:43:16Z)
• Unexpected but informative: What fixation-related potentials tell us about the processing of confusing program code [15.510640091254887]
  We analyze the online processing of program code patterns that are ambiguous to programmers, but not the computer. Relative to unambiguous counterparts in program code, atoms of confusion elicit a late frontal positivity with a duration of about 400 to 700 ms. We take these data to suggest that the brain engages similar neurocognitive mechanisms in response to unexpected and informative inputs in program code and in natural language.
  arXiv  Detail & Related papers  (2024-12-13T12:38:10Z)
• Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers [0.0]
  This study presents a thorough evaluation of leading programming assistants, including ChatGPT, Gemini(Bard AI), AlphaCode, and GitHub Copilot. It emphasizes the need for ethical developmental practices to actualize AI models' full potential.
  arXiv  Detail & Related papers  (2024-11-14T06:40:55Z)
• OpenHands: An Open Platform for AI Software Developers as Generalist Agents [109.8507367518992]
  We introduce OpenHands, a platform for the development of AI agents that interact with the world in similar ways to a human developer. We describe how the platform allows for the implementation of new agents, safe interaction with sandboxed environments for code execution, and incorporation of evaluation benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T17:50:43Z)
• CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation [58.84212778960507]
  We propose CodeGRAG, a Graphical Retrieval Augmented Code Generation framework to enhance the performance of LLMs. CodeGRAG builds the graphical view of code blocks based on the control flow and data flow of them to fill the gap between programming languages and natural language. Various experiments and ablations are done on four datasets including both the C++ and python languages to validate the hard meta-graph prompt, the soft prompting technique, and the effectiveness of the objectives for pretrained GNN expert.
  arXiv  Detail & Related papers  (2024-05-03T02:48:55Z)
• NExT: Teaching Large Language Models to Reason about Code Execution [50.93581376646064]
  Large language models (LLMs) of code are typically trained on the surface textual form of programs. We propose NExT, a method to teach LLMs to inspect the execution traces of programs and reason about their run-time behavior.
  arXiv  Detail & Related papers  (2024-04-23T01:46:32Z)
• Genetic Auto-prompt Learning for Pre-trained Code Intelligence Language Models [54.58108387797138]
  We investigate the effectiveness of prompt learning in code intelligence tasks. Existing automatic prompt design methods are very limited to code intelligence tasks. We propose Genetic Auto Prompt (GenAP) which utilizes an elaborate genetic algorithm to automatically design prompts.
  arXiv  Detail & Related papers  (2024-03-20T13:37:00Z)
• Exploring Prompt Engineering Practices in the Enterprise [3.7882262667445734]
  A prompt is a natural language instruction designed to elicit certain behaviour or output from a model. For complex tasks and tasks with specific requirements, prompt design is not trivial. We analyze sessions of prompt editing behavior, categorizing the parts of prompts users iterated on and the types of changes they made.
  arXiv  Detail & Related papers  (2024-03-13T20:32:32Z)
• Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilot [6.042269506496206]
  We investigate the influence of eight prompt features on the style and the content of prompts, on the level of correctness, complexity, size, and similarity to the developers' code. Results show how some prompt features, such as the presence of examples and the summary of the purpose of the method, can significantly influence the quality of the result.
  arXiv  Detail & Related papers  (2024-02-13T12:58:53Z)
• PwR: Exploring the Role of Representations in Conversational Programming [17.838776812138626]
  We introduce Programming with Representations (PwR), an approach that uses representations to convey the system's understanding back to the user in natural language. We find that representations significantly improve understandability, and instilled a sense of agency among our participants.
  arXiv  Detail & Related papers  (2023-09-18T05:38:23Z)
• ChatDev: Communicative Agents for Software Development [84.90400377131962]
  ChatDev is a chat-powered software development framework in which specialized agents are guided in what to communicate. These agents actively contribute to the design, coding, and testing phases through unified language-based communication.
  arXiv  Detail & Related papers  (2023-07-16T02:11:34Z)
• A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges [23.467373994306524]
  In practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. To understand developers' practices while using these tools, we administered a survey to a large population of developers. We found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements.
  arXiv  Detail & Related papers  (2023-03-30T03:21:53Z)
• A Case Study in Engineering a Conversational Programming Assistant's Persona [72.47187215119664]
  Conversational capability was achieved by using an existing code-fluent Large Language Model. A discussion of the evolution of the prompt provides a case study in how to coax an existing foundation model to behave in a desirable manner for a particular application.
  arXiv  Detail & Related papers  (2023-01-13T14:48:47Z)
• A Conversational Paradigm for Program Synthesis [110.94409515865867]
  We propose a conversational program synthesis approach via large language models. We train a family of large language models, called CodeGen, on natural language and programming language data. Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm.
  arXiv  Detail & Related papers  (2022-03-25T06:55:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.