Improving Code LLM Robustness to Prompt Perturbations via Layer-Aware Model Editing

• URL: http://arxiv.org/abs/2507.16407v1
• Date: Tue, 22 Jul 2025 09:57:55 GMT
• Title: Improving Code LLM Robustness to Prompt Perturbations via Layer-Aware Model Editing
• Authors: Shuhan Liu, Xing Hu, Kerui Huang, Xiaohu Yang, David Lo, Xin Xia,
• Abstract summary: Large language models (LLMs) are highly sensitive to prompt perturbations.<n>We introduce CREME, a novel approach that enhances LLM robustness through targeted parameter updates.<n> Experimental results show that CREME improves Pass@1 accuracy by 63% on perturbed prompts.
• Score: 13.099973383252452
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities in code generation, where the natural language prompt plays a crucial role in conveying user intent to the model. However, prior studies have shown that LLMs are highly sensitive to prompt perturbations. Minor modifications in wording, syntax, or formatting can significantly reduce the functional correctness of generated code. As perturbations frequently occur in real-world scenarios, improving the robustness of LLMs to prompt perturbations is essential for ensuring reliable performance in practical code generation. In this paper, we introduce CREME (Code Robustness Enhancement via Model Editing), a novel approach that enhances LLM robustness through targeted parameter updates. CREME first identifies robustness-sensitive layers by comparing hidden states between an original prompt and its perturbed variant. Then, it performs lightweight parameter editing at the identified layer to reduce performance degradation. We evaluate CREME on two widely used code generation benchmarks (HumanEval and MBPP) along with their perturbed counterparts. Experimental results show that CREME improves Pass@1 accuracy by 63% on perturbed prompts while maintaining stable performance on clean inputs, with accuracy deviations within 1%. Further analysis reveals that robustness-sensitive layers are primarily concentrated in the middle and deeper layers of the network, and their locations vary across different model architectures. These insights provide a valuable foundation for developing future robustness-oriented editing strategies.

Related papers

• Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attacks [8.901793877849155]
  Robustness of Prompting (RoP) is a novel prompting strategy specifically designed to enhance the robustness of Large Language Models (LLMs)<n>RoP applies diverse perturbation methods to generate adversarial examples, which are then used to construct prompts that automatically correct input errors.<n>In the Guidance stage, RoP generates an optimal guidance prompting based on the corrected input, steering the model toward more robust and accurate inferences.
  arXiv  Detail & Related papers  (2025-06-04T07:13:27Z)
• Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding [51.711605076319216]
  Diffusion-based large language models (Diffusion LLMs) have shown promise for non-autoregressive text generation with parallel decoding capabilities.<n>We introduce a novel block-wise approximate KV Cache mechanism tailored for bidirectional diffusion models, enabling cache reuse with negligible performance drop.<n>We propose a confidence-aware parallel decoding strategy that selectively decodes tokens exceeding a confidence threshold, mitigating dependency violations and maintaining generation quality.
  arXiv  Detail & Related papers  (2025-05-28T17:39:15Z)
• CodeCrash: Stress Testing LLM Reasoning under Structural and Semantic Perturbations [36.60702578561009]
  Large Language Models (LLMs) have recently demonstrated strong capabilities in code-related tasks, yet their robustness in code comprehension and reasoning remains insufficiently explored.<n>We present CodeCrash, a comprehensive stress-testing benchmark comprising 1,279 questions from two established datasets.<n>We systematically evaluate 17 LLMs across input and output prediction tasks using direct and Chain-of-Thought prompting approaches.
  arXiv  Detail & Related papers  (2025-04-19T00:40:28Z)
• Enhancing LLM Robustness to Perturbed Instructions: An Empirical Study [8.827173113748701]
  We study character- and word-level edits of task-specific instructions, which substantially degrade downstream performance.<n>We find that, on average, self-denoising achieves substantially higher performance gains than alternative strategies.
  arXiv  Detail & Related papers  (2025-04-03T16:17:56Z)
• Model Hemorrhage and the Robustness Limits of Large Language Models [119.46442117681147]
  Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment.<n>We define this phenomenon as model hemorrhage - performance decline caused by parameter alterations and architectural changes.
  arXiv  Detail & Related papers  (2025-03-31T10:16:03Z)
• Forget What You Know about LLMs Evaluations - LLMs are Like a Chameleon [11.753349115726952]
  Large language models (LLMs) often appear to excel on public benchmarks, but these high scores may mask an overreliance on dataset-specific surface cues.<n>We introduce the Chameleon Benchmark Overfit Detector (C-BOD), a meta-evaluation framework that distorts benchmark prompts.<n>By rephrasing inputs while preserving semantic content and labels, C-BOD exposes whether a model's performance is driven by memorized patterns.
  arXiv  Detail & Related papers  (2025-02-11T10:43:36Z)
• Self-Evolving Critique Abilities in Large Language Models [59.861013614500024]
  This paper explores enhancing critique abilities of Large Language Models (LLMs)<n>We introduce SCRIT, a framework that trains LLMs with self-generated data to evolve their critique abilities.<n>Our analysis reveals that SCRIT's performance scales positively with data and model size.
  arXiv  Detail & Related papers  (2025-01-10T05:51:52Z)
• Less is More: Towards Green Code Large Language Models via Unified Structural Pruning [27.428983811427827]
  We propose Flab-Pruner, an innovative unified structural pruning method that combines vocabulary, layer, and Feed-Forward Network (FFN) pruning.<n>The results demonstrate that Flab-Pruner retains 97% of the original performance after pruning 22% of the parameters and achieves the same or even better performance after post-training.
  arXiv  Detail & Related papers  (2024-12-20T14:13:09Z)
• On the Worst Prompt Performance of Large Language Models [93.13542053835542]
  Performance of large language models (LLMs) is acutely sensitive to the phrasing of prompts. We introduce RobustAlpacaEval, a new benchmark that consists of semantically equivalent case-level queries. Experiments on RobustAlpacaEval with ChatGPT and six open-source LLMs from the Llama, Mistral, and Gemma families uncover substantial variability in model performance.
  arXiv  Detail & Related papers  (2024-06-08T13:40:38Z)
• FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping [49.66872823080736]
  Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent achieving remarkable success in language understanding and generation. To mitigate overload incurred during generation, several early-exit and layer-dropping strategies have been proposed. We propose FFN-SkipLLM, which is an input-adaptive feed-forward skipping strategy.
  arXiv  Detail & Related papers  (2024-04-05T02:35:43Z)
• Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE [62.13435256279566]
  Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks. However, their large size makes their inference slow and computationally expensive. We show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer.
  arXiv  Detail & Related papers  (2023-10-28T04:07:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.