Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm

• URL: http://arxiv.org/abs/2509.24637v2
• Date: Mon, 13 Oct 2025 04:21:45 GMT
• Title: Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm
• Abstract summary: We introduce Instruction-aware Fill-in-the-Middle (IFIM) to enhance code completion models.<n>IFIM incorporates an explicit instruction section into the input, enabling the model to learn from triplets.<n>Results show IFIM significantly improves instruction-following capabilities, boosting the Pass@1 score from 84.6% to 93.6%.
• Score: 14.429414087640886
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have significantly advanced code completion, yet they often fail when the developer's intent is underspecified in the code context. To address this, developers usually add natural language instructions (e.g., comments) into the code context to clarify their intent. However, existing code LLMs applied for code completion systems merely undergo a fill-in-the-middle (FIM) pre-training, which struggles to leverage this information effectively due to the lack of instruction-like training data. Existing instruction-tuning techniques, improving instruction-following in general code generation, paradoxically degrades FIM performance, forcing a trade-off between instruction-following and infilling capabilities. To address this gap, we introduce Instruction-aware Fill-in-the-Middle (IFIM), an instruction-tuning method specifically designed to enhance FIM code completion models. IFIM extends the conventional FIM training objective by incorporating an explicit instruction section into the input, enabling the model to learn from (prefix, instruction, suffix) triplets. This approach allows the model to effectively leverage developer-provided directives while preserving its core completion abilities when no instructions are present. To facilitate this, we constructed a large-scale dataset by using GPT-4o to generate concise, intent-focused instructions for code infilling examples. We evaluated IFIM by applying it to two popular base models, Deepseek-Coder and Qwen2.5-Coder, on the benchmarks derived from HumanEval-infilling and RepoMasterEval. The results demonstrate that IFIM significantly improves instruction-following capabilities, boosting the Pass@1 score from 84.6% to 93.6%. Moreover, this enhancement does not compromise the models' original performance on FIM code completion tasks with no instructions provided.

Related papers

• Lost in the Flow with Code Talkers: Unveiling the Instruction-Tuning Tax of Large Language Models in Code Tasks [6.645779775557948]
  We conduct the first empirical study that uncovers a key trade-off caused by instruction tuning across programming modes.<n>Our results show that instruction tuning is not a free lunch: although instruction-tuned models are more capable of following instructions, these gains often come at the cost of weaker infilling performance.
  arXiv  Detail & Related papers  (2026-06-07T15:24:30Z)
• Evaluating and Achieving Controllable Code Completion in Code LLM [89.64782747840225]
  We present the first instruction-guided code completion benchmark, Controllable Code Completion Benchmark (C3-Bench)<n>We reveal substantial gaps in instruction-following capabilities between open-source and advanced proprietary models during code completion tasks.<n>The resulting model, Qwen2.5-Coder-C3, achieves state-of-the-art performance on C3-Bench.
  arXiv  Detail & Related papers  (2026-01-22T11:40:04Z)
• Effective Code Membership Inference for Code Completion Models via Adversarial Prompts [17.428753624187717]
  Membership inference attacks (MIAs) on code completion models offer an effective way to assess privacy risks.<n>We propose AdvPrompt-MIA, a method specifically designed for code completion models, combining code-specific adversarial perturbations with deep learning.<n>We conduct comprehensive evaluations on widely adopted models, such as Code Llama 7B, over the APPS and HumanEval benchmarks.
  arXiv  Detail & Related papers  (2025-11-19T04:30:54Z)
• Learning Decomposed Contextual Token Representations from Pretrained and Collaborative Signals for Generative Recommendation [17.061613097917217]
  We propose a unified framework that preserves pretrained semantics while enhancing the adaptability of token embeddings.<n>Experiments on three real-world datasets demonstrate that DECOR consistently outperforms state-of-the-art baselines in recommendation performance.
  arXiv  Detail & Related papers  (2025-08-22T18:50:38Z)
• CodeBoost: Boosting Code LLMs by Squeezing Knowledge from Code Snippets with RL [28.43882967593511]
  Code large language models (LLMs) have become indispensable tools for building efficient and automated coding pipelines.<n>Existing models are typically post-trained using reinforcement learning (RL) from general-purpose LLMs using "human instruction-final answer" pairs.<n>We propose CodeBoost, a framework that enhances code LLMs purely from code snippets, without relying on human-annotated instructions.
  arXiv  Detail & Related papers  (2025-08-07T10:31:24Z)
• Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models [14.927052028174744]
  Infinity-Instruct is a high-quality instruction dataset designed to enhance both foundational and chat capabilities of Large Language Models.<n>We empirically evaluate Infinity-Instruct by fine-tuning several open-source models, including Mistral, LLaMA, Qwen, and Yi, and observe substantial performance gains across both foundational and instruction following benchmarks.
  arXiv  Detail & Related papers  (2025-06-09T06:37:15Z)
• Training with Pseudo-Code for Instruction Following [4.7188893422904]
  We take inspiration from recent work that suggests that models may follow instructions better when they are expressed in pseudo-code.<n>We propose fine-tuning Large Language Models with instruction-tuning data that additionally includes instructions re-expressed in pseudo-code.<n>We conduct rigorous experiments with $5$ different models and find that not only do models follow instructions better when trained with pseudo-code, they also retain their capabilities on the other tasks related to mathematical and common sense reasoning.
  arXiv  Detail & Related papers  (2025-05-23T15:14:29Z)
• Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models [54.14602121129874]
  We introduce AutoIF, the first scalable and reliable method for automatically generating instruction-following training data. AutoIF transforms the validation of instruction-following data quality into code verification.
  arXiv  Detail & Related papers  (2024-06-19T13:29:53Z)
• CodecLM: Aligning Language Models with Tailored Synthetic Data [51.59223474427153]
  We introduce CodecLM, a framework for adaptively generating high-quality synthetic data for instruction-following abilities. We first encode seed instructions into metadata, which are concise keywords generated on-the-fly to capture the target instruction distribution. We also introduce Self-Rubrics and Contrastive Filtering during decoding to tailor data-efficient samples.
  arXiv  Detail & Related papers  (2024-04-08T21:15:36Z)
• Contrastive Instruction Tuning [61.97704869248903]
  We propose Contrastive Instruction Tuning to maximize the similarity between semantically equivalent instruction-instance pairs. Experiments on the PromptBench benchmark show that CoIN consistently improves LLMs' robustness to unseen instructions with variations across character, word, sentence, and semantic levels by an average of +2.5% in accuracy.
  arXiv  Detail & Related papers  (2024-02-17T00:09:32Z)
• From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning [63.63840740526497]
  We investigate how instruction tuning adjusts pre-trained models with a focus on intrinsic changes. The impact of instruction tuning is then studied by comparing the explanations derived from the pre-trained and instruction-tuned models. Our findings reveal three significant impacts of instruction tuning.
  arXiv  Detail & Related papers  (2023-09-30T21:16:05Z)
• Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection [70.28425745910711]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following. This capability brings with it the risk of prompt injection attacks. We evaluate the robustness of instruction-following LLMs against such attacks.
  arXiv  Detail & Related papers  (2023-08-17T06:21:50Z)
• CodeT5+: Open Code Large Language Models for Code Understanding and Generation [72.1638273937025]
  Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. CodeT5+ is a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning.
  arXiv  Detail & Related papers  (2023-05-13T14:23:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.