Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification

• URL: http://arxiv.org/abs/2508.01523v1
• Date: Sat, 02 Aug 2025 23:52:49 GMT
• Title: Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification
• Authors: Ningzhi Tang, Emory Smith, Yu Huang, Collin McMillan, Toby Jia-Jun Li,
• Abstract summary: This paper presents a study of using large language models (LLMs) in modifying existing code.<n>"prompting" serves as the primary interface for developers to communicate intents to LLMs.<n>This study investigates two prompting strategies for LLM-assisted code modification.
• Score: 10.964060011243234
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a study of using large language models (LLMs) in modifying existing code. While LLMs for generating code have been widely studied, their role in code modification remains less understood. Although "prompting" serves as the primary interface for developers to communicate intents to LLMs, constructing effective prompts for code modification introduces challenges different from generation. Prior work suggests that natural language summaries may help scaffold this process, yet such approaches have been validated primarily in narrow domains like SQL rewriting. This study investigates two prompting strategies for LLM-assisted code modification: Direct Instruction Prompting, where developers describe changes explicitly in free-form language, and Summary-Mediated Prompting, where changes are made by editing the generated summaries of the code. We conducted an exploratory study with 15 developers who completed modification tasks using both techniques across multiple scenarios. Our findings suggest that developers followed an iterative workflow: understanding the code, localizing the edit, and validating outputs through execution or semantic reasoning. Each prompting strategy presented trade-offs: direct instruction prompting was more flexible and easier to specify, while summary-mediated prompting supported comprehension, prompt scaffolding, and control. Developers' choice of strategy was shaped by task goals and context, including urgency, maintainability, learning intent, and code familiarity. These findings highlight the need for more usable prompt interactions, including adjustable summary granularity, reliable summary-code traceability, and consistency in generated summaries.

Related papers

• IFEvalCode: Controlled Code Generation [69.28317223249358]
  The paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs.<n>The authors present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages.
  arXiv  Detail & Related papers  (2025-07-30T08:08:48Z)
• Post-Incorporating Code Structural Knowledge into LLMs via In-Context Learning for Code Translation [10.77747590700758]
  Large language models (LLMs) have achieved significant advancements in software mining.<n> handling the syntactic structure of source code remains a challenge.<n>This paper employs incontext learning (ICL) to integrate code structural knowledge into pre-trained LLMs.
  arXiv  Detail & Related papers  (2025-03-28T10:59:42Z)
• Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories [11.06441376653589]
  This study presents the first empirical analysis of prompt evolution in LLM-integrated software development.<n>We analyzed 1,262 prompt changes across 243 GitHub repositories to investigate the patterns and frequencies of prompt changes.<n>Our findings show that developers primarily evolve prompts through additions and modifications, with most changes occurring during feature development.
  arXiv  Detail & Related papers  (2024-12-23T05:41:01Z)
• CodeEditorBench: Evaluating Code Editing Capability of Large Language Models [49.387195629660994]
  Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability.<n>We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks.<n>We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks.
  arXiv  Detail & Related papers  (2024-04-04T15:49:49Z)
• Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs [65.2379940117181]
  We introduce code prompting, a chain of prompts that transforms a natural language problem into code. We find that code prompting exhibits a high-performance boost for multiple LLMs. Our analysis of GPT 3.5 reveals that the code formatting of the input problem is essential for performance improvement.
  arXiv  Detail & Related papers  (2024-01-18T15:32:24Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• A Prompt Learning Framework for Source Code Summarization [19.24919436211323]
  This paper proposes an effective prompt learning framework for code summarization called PromptCS.<n>PromptCS trains a prompt agent that can generate continuous prompts to unleash the potential for large language models in code summarization.
  arXiv  Detail & Related papers  (2023-12-26T14:37:55Z)
• InstructCoder: Instruction Tuning Large Language Models for Code Editing [26.160498475809266]
  We explore the use of Large Language Models (LLMs) to edit code based on user instructions. InstructCoder is the first instruction-tuning dataset designed to adapt LLMs for general-purpose code editing. Our findings reveal that open-source LLMs fine-tuned on InstructCoder can significantly enhance the accuracy of code edits.
  arXiv  Detail & Related papers  (2023-10-31T10:15:35Z)
• Fixing Large Language Models' Specification Misunderstanding for Better Code Generation [13.494822086550604]
  muFiX is a novel prompting technique to improve the code generation performance of large language models (LLMs)<n>It first exploits test case analysis to obtain specification understanding and enables a self-improvement process.<n>muFiX further fixes the specification understanding towards the direction reducing the gap between the provided understanding and the actual understanding.
  arXiv  Detail & Related papers  (2023-09-28T02:58:07Z)
• Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning [34.006227676170504]
  This study investigates the feasibility of utilizing large language models (LLMs) to generate comments that can fulfill developers' diverse intents. Experiments on two large-scale datasets demonstrate the rationale of our insights.
  arXiv  Detail & Related papers  (2023-04-22T12:26:24Z)
• Guiding Large Language Models via Directional Stimulus Prompting [114.84930073977672]
  We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model to generate an auxiliary directional stimulus prompt for each input instance.
  arXiv  Detail & Related papers  (2023-02-22T17:44:15Z)
• OpenPrompt: An Open-source Framework for Prompt-learning [59.17869696803559]
  We present OpenPrompt, a unified easy-to-use toolkit to conduct prompt-learning over PLMs. OpenPrompt is a research-friendly framework that is equipped with efficiency, modularity, and extendibility.
  arXiv  Detail & Related papers  (2021-11-03T03:31:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.