Understanding Prompt Programming Tasks and Questions

• URL: http://arxiv.org/abs/2507.17264v1
• Date: Wed, 23 Jul 2025 07:01:44 GMT
• Title: Understanding Prompt Programming Tasks and Questions
• Abstract summary: We develop a taxonomy of 25 tasks prompt programmers do and 51 questions they ask, measuring the importance of each task and question.<n>We find that prompt programming is not well-supported: all tasks are done manually, and 16 of the 51 questions -- including a majority of the most important ones -- remain unanswered.
• Score: 19.414347044995644
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompting foundation models (FMs) like large language models (LLMs) have enabled new AI-powered software features (e.g., text summarization) that previously were only possible by fine-tuning FMs. Now, developers are embedding prompts in software, known as prompt programs. The process of prompt programming requires the developer to make many changes to their prompt. Yet, the questions developers ask to update their prompt is unknown, despite the answers to these questions affecting how developers plan their changes. With the growing number of research and commercial prompt programming tools, it is unclear whether prompt programmers' needs are being adequately addressed. We address these challenges by developing a taxonomy of 25 tasks prompt programmers do and 51 questions they ask, measuring the importance of each task and question. We interview 16 prompt programmers, observe 8 developers make prompt changes, and survey 50 developers. We then compare the taxonomy with 48 research and commercial tools. We find that prompt programming is not well-supported: all tasks are done manually, and 16 of the 51 questions -- including a majority of the most important ones -- remain unanswered. Based on this, we outline important opportunities for prompt programming tools.

Related papers

• Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks [49.47167381167488]
  We quantify which types of information most affect task success and which questions elicit useful responses from simulated users.<n>Using Shapley attribution and distributional comparisons, we identify two key properties of effective clarification: task relevance (which information predicts success) and user answerability.<n>We operationalize these properties as multi-stage reinforcement learning rewards to train CLARITI, an 8B- parameter clarification module, that matches GPT-5's resolution rate on underspecified issues.
  arXiv  Detail & Related papers  (2026-04-16T05:11:02Z)
• Automatically Generating Questions About Scratch Programs [10.726203107125315]
  We propose a set of 30 different questions for Scratch code covering an established program comprehension model.<n>We extend the LitterBox static analysis tool to automatically generate corresponding questions for a given Scratch program.<n>On a dataset of 600,913 projects we generated 54,118,694 questions automatically.
  arXiv  Detail & Related papers  (2025-10-13T17:34:12Z)
• An Empirically-grounded tool for Automatic Prompt Linting and Repair: A Case Study on Bias, Vulnerability, and Optimization in Developer Prompts [2.6254294912127127]
  We introduce PromptDoctor, a tool specifically designed to detect and correct issues of Dev Prompts.<n>In our analysis of 2,173 Dev Prompts, selected as a representative sample of 40,573 Dev Prompts, we found that 3.46% contained one or more forms of bias.<n> PromptDoctor de-biased 68.29% of the biased Dev Prompts, hardened 41.81% of the vulnerable Dev Prompts, and improved the performance of 37.1% sub-optimal Dev Prompts.
  arXiv  Detail & Related papers  (2025-01-21T22:24:03Z)
• Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution [3.768737590492549]
  We analyze 433 developer-ChatGPT conversations within GitHub issue threads to examine the impact of prompt knowledge gaps and conversation styles on issue resolution.<n>We find that ineffective conversations contain knowledge gaps in 44.6% of prompts, compared to only 12.6% in effective ones.
  arXiv  Detail & Related papers  (2025-01-20T19:41:42Z)
• Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts [12.979519344826748]
  We argue that some prompts are programs and that the development of prompts is a distinct phenomenon in programming known as "prompt programming"<n>We develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development.
  arXiv  Detail & Related papers  (2024-09-19T03:55:49Z)
• I Could've Asked That: Reformulating Unanswerable Questions [89.93173151422636]
  We evaluate open-source and proprietary models for reformulating unanswerable questions. GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time, respectively. We publicly release the benchmark and the code to reproduce the experiments.
  arXiv  Detail & Related papers  (2024-07-24T17:59:07Z)
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions [72.56339136017759]
  We introduce BigCodeBench, a benchmark that challenges Large Language Models (LLMs) to invoke multiple function calls as tools from 139 libraries and 7 domains for 1,140 fine-grained tasks.<n>Our evaluation shows that LLMs are not yet capable of following complex instructions to use function calls precisely, with scores up to 60%, significantly lower than the human performance of 97%.<n>We propose a natural-language-oriented variant of BigCodeBench, BigCodeBench-Instruct, that automatically transforms the original docstrings into short instructions only with essential information.
  arXiv  Detail & Related papers  (2024-06-22T15:52:04Z)
• Answering Ambiguous Questions with a Database of Questions, Answers, and Revisions [95.92276099234344]
  We present a new state-of-the-art for answering ambiguous questions that exploits a database of unambiguous questions generated from Wikipedia. Our method improves performance by 15% on recall measures and 10% on measures which evaluate disambiguating questions from predicted outputs.
  arXiv  Detail & Related papers  (2023-08-16T20:23:16Z)
• Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning [43.83422798569986]
  Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and reliable form of assessment. To date, the task of crafting high-quality distractors has largely remained a labor-intensive process for teachers and learning content designers. We propose a simple, in-context learning-based solution for automated distractor and corresponding feedback message generation.
  arXiv  Detail & Related papers  (2023-08-07T01:03:04Z)
• Employing Deep Learning and Structured Information Retrieval to Answer Clarification Questions on Bug Reports [3.462843004438096]
  We propose a novel approach that uses CodeT5 in combination with Lucene to recommend answers to follow-up questions. We evaluate our recommended answers with the manually annotated answers using similarity metrics like Normalized Smooth BLEU Score, METEOR, Word Mover's Distance, and Semantic Similarity.
  arXiv  Detail & Related papers  (2023-04-24T23:29:14Z)
• A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges [23.467373994306524]
  In practice, developers do not accept AI programming assistants' initial suggestions at a high frequency. To understand developers' practices while using these tools, we administered a survey to a large population of developers. We found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax. We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements.
  arXiv  Detail & Related papers  (2023-03-30T03:21:53Z)
• Brain-inspired Search Engine Assistant based on Knowledge Graph [53.89429854626489]
  DeveloperBot is a brain-inspired search engine assistant named on knowledge graph. It constructs a multi-layer query graph by splitting a complex multi-constraint query into several ordered constraints. It then models the constraint reasoning process as subgraph search process inspired by the spreading activation model of cognitive science.
  arXiv  Detail & Related papers  (2020-12-25T06:36:11Z)
• Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning [56.771557756836906]
  We present a novel method that automatically learns a retrieval model alternately with the programmer from weak supervision. Our system leads to state-of-the-art performance on a large-scale task for complex question answering over knowledge bases.
  arXiv  Detail & Related papers  (2020-10-29T18:28:16Z)
• Inquisitive Question Generation for High Level Text Comprehension [60.21497846332531]
  We introduce INQUISITIVE, a dataset of 19K questions that are elicited while a person is reading through a document. We show that readers engage in a series of pragmatic strategies to seek information. We evaluate question generation models based on GPT-2 and show that our model is able to generate reasonable questions.
  arXiv  Detail & Related papers  (2020-10-04T19:03:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.