Understanding Prompt Management in GitHub Repositories: A Call for Best Practices

• URL: http://arxiv.org/abs/2509.12421v1
• Date: Mon, 15 Sep 2025 20:18:22 GMT
• Title: Understanding Prompt Management in GitHub Repositories: A Call for Best Practices
• Abstract summary: The rapid adoption of foundation models has given rise to promptware, i.e., software built using natural language prompts.<n>In this study, we perform an empirical analysis of 24,800 open-source prompts from 92 GitHub repositories to investigate prompt management practices and quality attributes.
• Score: 11.853151610220111
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid adoption of foundation models (e.g., large language models) has given rise to promptware, i.e., software built using natural language prompts. Effective management of prompts, such as organization and quality assurance, is essential yet challenging. In this study, we perform an empirical analysis of 24,800 open-source prompts from 92 GitHub repositories to investigate prompt management practices and quality attributes. Our findings reveal critical challenges such as considerable inconsistencies in prompt formatting, substantial internal and external prompt duplication, and frequent readability and spelling issues. Based on these findings, we provide actionable recommendations for developers to enhance the usability and maintainability of open-source prompts within the rapidly evolving promptware ecosystem.

Related papers

• A Study of Cursorrules Files in GitHub Open Source Projects [5.527667715674739]
  We conduct an empirical study of configuration prompt files in Cursor, a widely used AI-assisted code editor.<n>We analyze over 12,110.cursorrules files from 11,427 GitHub repositories to characterize their distribution, evolution, and maintenance.
  arXiv  Detail & Related papers  (2026-08-11T08:08:49Z)
• Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering [3.788792284009516]
  We present Prompt-with-Me, a practical solution for structured prompt management embedded directly in the development environment.<n>The system automatically classifies prompts using a four-dimensional taxonomy encompassing intent, author role, software development lifecycle stage, and prompt type.
  arXiv  Detail & Related papers  (2025-09-21T14:24:37Z)
• Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution [3.768737590492549]
  We analyze 433 developer-ChatGPT conversations within GitHub issue threads to examine the impact of prompt knowledge gaps and conversation styles on issue resolution.<n>We find that ineffective conversations contain knowledge gaps in 44.6% of prompts, compared to only 12.6% in effective ones.
  arXiv  Detail & Related papers  (2025-01-20T19:41:42Z)
• Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories [11.06441376653589]
  This study presents the first empirical analysis of prompt evolution in LLM-integrated software development.<n>We analyzed 1,262 prompt changes across 243 GitHub repositories to investigate the patterns and frequencies of prompt changes.<n>Our findings show that developers primarily evolve prompts through additions and modifications, with most changes occurring during feature development.
  arXiv  Detail & Related papers  (2024-12-23T05:41:01Z)
• Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration [64.19431011897515]
  This paper presents Alibaba LingmaAgent, a novel Automated Software Engineering method designed to comprehensively understand and utilize whole software repositories for issue resolution.<n>Our approach introduces a top-down method to condense critical repository information into a knowledge graph, reducing complexity, and employs a Monte Carlo tree search based strategy.<n>In production deployment and evaluation at Alibaba Cloud, LingmaAgent automatically resolved 16.9% of in-house issues faced by development engineers, and solved 43.3% of problems after manual intervention.
  arXiv  Detail & Related papers  (2024-06-03T15:20:06Z)
• Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression [14.028851291935625]
  Existing attempts to compress lengthy prompts lead to substandard results in terms of readability/interpretability of the compressed prompt. We propose PromptSAW: Prompt compresSion via Relation AWare graphs, an effective strategy for prompt compression over task-agnostic and task-aware prompts. We also propose GSM8K-aug, i.e., an extended version of the existing GSM8K benchmark for task-agnostic prompts in order to provide a comprehensive evaluation platform.
  arXiv  Detail & Related papers  (2024-03-30T23:07:58Z)
• Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilot [6.042269506496206]
  We investigate the influence of eight prompt features on the style and the content of prompts, on the level of correctness, complexity, size, and similarity to the developers' code. Results show how some prompt features, such as the presence of examples and the summary of the purpose of the method, can significantly influence the quality of the result.
  arXiv  Detail & Related papers  (2024-02-13T12:58:53Z)
• PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization [60.00631098364391]
  PromptAgent is an optimization method that crafts expert-level prompts equivalent in quality to those handcrafted by experts. Inspired by human-like trial-and-error exploration, PromptAgent induces precise expert-level insights and in-depth instructions. We apply PromptAgent to 12 tasks spanning three practical domains.
  arXiv  Detail & Related papers  (2023-10-25T07:47:01Z)
• InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding [51.48361798508375]
  We develop an information-theoretic framework that formulates soft prompt tuning as maximizing mutual information between prompts and other model parameters. We show that InfoPrompt can significantly accelerate the convergence of the prompt tuning and outperform traditional prompt tuning methods.
  arXiv  Detail & Related papers  (2023-06-08T04:31:48Z)
• PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts [76.18347405302728]
  This study uses a plethora of adversarial textual attacks targeting prompts across multiple levels: character, word, sentence, and semantic. The adversarial prompts are then employed in diverse tasks including sentiment analysis, natural language inference, reading comprehension, machine translation, and math problem-solving. Our findings demonstrate that contemporary Large Language Models are not robust to adversarial prompts.
  arXiv  Detail & Related papers  (2023-06-07T15:37:00Z)
• Visual-Language Prompt Tuning with Knowledge-guided Context Optimization [96.27531485377871]
  Representative CoOp-based work combines the learnable textual tokens with the class tokens to obtain specific textual knowledge. We introduce a novel Knowledge-guided Context Optimization (KgCoOp) to enhance the generalization ability of the learnable prompt for unseen classes.
  arXiv  Detail & Related papers  (2023-03-23T14:04:23Z)
• Demystifying Prompts in Language Models via Perplexity Estimation [109.59105230163041]
  Performance of a prompt is coupled with the extent to which the model is familiar with the language it contains. We show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task.
  arXiv  Detail & Related papers  (2022-12-08T02:21:47Z)
• MetaPrompting: Learning to Learn Better Prompts [52.914694884515534]
  We propose a new soft prompting method called MetaPrompting, which adopts the well-recognized model-agnostic meta-learning algorithm. Extensive experiments show MetaPrompting brings significant improvement on four different datasets.
  arXiv  Detail & Related papers  (2022-09-23T09:01:05Z)
• RLPrompt: Optimizing Discrete Text Prompts With Reinforcement Learning [84.75064077323098]
  This paper proposes RLPrompt, an efficient discrete prompt optimization approach with reinforcement learning (RL) RLPrompt is flexibly applicable to different types of LMs, such as masked gibberish (e.g., grammaBERT) and left-to-right models (e.g., GPTs) Experiments on few-shot classification and unsupervised text style transfer show superior performance over a wide range of existing finetuning or prompting methods.
  arXiv  Detail & Related papers  (2022-05-25T07:50:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.