Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills

• URL: http://arxiv.org/abs/2605.29354v1
• Date: Thu, 28 May 2026 04:48:45 GMT
• Title: Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills
• Abstract summary: This paper introduces emphNeutral Prompting Attack (NPA)<n>NPA is a stealthy attack paradigm in which semantically benign instructions, such as encouraging imagination, increase package hallucination propensity without containing explicit malicious intent.<n>Our results show that NPA increases both emphHallucination ASR and emphPip Install ASR, changes the distribution of hallucinated package names, and evades existing static-analysis, LLM-based, and agent-based Skill defenses.
• Score: 15.148439025792017
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-powered coding agents increasingly participate in software development workflows by generating code, selecting dependencies, and producing package installation commands. This creates a new software supply chain risk: when an agent hallucinates a non-existent package, an attacker may register the hallucinated name and later compromise users who install it. Existing package hallucination attacks and defenses primarily focus on naturally occurring hallucinations, targeted dependency steering, or post-hoc package validation. In this paper, we introduce \emph{Neutral Prompting Attack} (NPA), a highly stealthy attack paradigm in which semantically benign instructions, such as encouraging imagination and exhaustiveness, increase package hallucination propensity without containing explicit malicious intent. Unlike targeted dependency steering, NPA does not specify an attacker-chosen package. Instead, it shifts the model's dependency generation behavior toward more speculative package names. We evaluate NPA across multiple coding-oriented LLMs and package hallucination benchmarks. Our results show that NPA increases both \emph{Hallucination ASR} and \emph{Pip Install ASR}, changes the distribution of hallucinated package names, and evades existing static-analysis, LLM-based, and agent-based Skill defenses. These findings reveal that harmless-looking prompts can covertly manipulate hallucination behavior and create downstream software supply chain risks.

Related papers

• Mitigating Package Hallucinations in Large Language Models via Model Editing [11.455817131025675]
  BOUND is a lightweight localized model editing framework for mitigating package hallucinations in large language models.<n>It first locates modules related to package hallucination through a risk-aware localization strategy, and then edits these modules with lightweight LoRA.<n> Experimental results show that BOUND effectively reduces package hallucinations while preserving valid package recommendations.
  arXiv  Detail & Related papers  (2026-07-02T11:27:01Z)
• Trust Me, Import This: Dependency Steering Attacks via Malicious Agent Skills [25.627983175254958]
  We introduce Dependency Steering, an attack paradigm in which a malicious Skill biases a coding agent toward an attacker-controlled package.<n>Our results show that persistent agent instructions form an underexplored software supply chain attack surface.
  arXiv  Detail & Related papers  (2026-05-10T15:13:38Z)
• LLM Ghostbusters: Surgical Hallucination Suppression via Adaptive Unlearning [12.855537727854975]
  We present Adaptive Unlearning, a post-deployment framework that surgically suppresses hallucinations while preserving general model utility.<n>We demonstrate that AU reduces package rates by 81%, corresponding to a substantial reduction in slopsquatting attack surface.<n>Our analysis shows that distributional changes are concentrated on package-related generations, leaving general coding behavior largely unaffected.
  arXiv  Detail & Related papers  (2026-05-01T19:20:31Z)
• SkillJect: Automating Stealthy Skill-Based Prompt Injection for Coding Agents with Trace-Driven Closed-Loop Refinement [120.52289344734415]
  We propose an automated framework for stealthy prompt injection tailored to agent skills.<n>The framework forms a closed loop with three agents: an Attack Agent that synthesizes injection skills under explicit stealth constraints, a Code Agent that executes tasks using the injected skills and an Evaluate Agent that logs action traces.<n>Our method consistently achieves high attack success rates under realistic settings.
  arXiv  Detail & Related papers  (2026-02-15T16:09:48Z)
• A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI [54.34738767990601]
  As Large Language Models become increasingly integrated into software engineering tasks, understanding and mitigating hallucination in code becomes essential.<n>We provide a systematic review of hallucination phenomena in code-oriented LLMs from four key perspectives.
  arXiv  Detail & Related papers  (2025-11-02T02:58:41Z)
• Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization [55.543583937522804]
  Multimodal Large Language Models (MLLMs) emerge as a unified interface to address a multitude of tasks.<n>Despite showcasing state-of-the-art results in many benchmarks, a long-standing issue is the tendency of MLLMs to hallucinate.<n>In this paper, we address the problem of hallucinations as an alignment problem, seeking to steer the MLLM so that it prefers generating content without hallucinations.
  arXiv  Detail & Related papers  (2025-08-27T18:02:04Z)
• JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering [73.962469626788]
  Jailbreak attacks against multimodal large language Models (MLLMs) are a significant research focus.<n>We propose JPS, underlineJailbreak MLLMs with collaborative visual underlinePerturbation and textual underlineSteering.
  arXiv  Detail & Related papers  (2025-08-07T07:14:01Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [78.78822033285938]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)
• Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities [11.868859925111561]
  Large Language Models (LLMs) have become an essential tool in the programmer's toolkit.<n>Their tendency to hallucinate code can be used by malicious actors to introduce vulnerabilities to broad swathes of the software supply chain.
  arXiv  Detail & Related papers  (2025-01-31T10:26:18Z)
• ShadowCode: Towards (Automatic) External Prompt Injection Attack against Code LLMs [56.46702494338318]
  This paper introduces a new attack paradigm: (automatic) external prompt injection against code-oriented large language models.<n>We propose ShadowCode, a simple yet effective method that automatically generates induced perturbations based on code simulation.<n>We evaluate our method across 13 distinct malicious objectives, generating 31 threat cases spanning three popular programming languages.
  arXiv  Detail & Related papers  (2024-07-12T10:59:32Z)
• We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs [3.515912713354746]
  Package hallucinations arise from fact-conflicting errors when generating code using Large Language Models.<n>This paper conducts a rigorous and comprehensive evaluation of package hallucinations across different programming languages.<n>We show that the average percentage of hallucinated packages is at least 5.2% for commercial models and 21.7% for open-source models.
  arXiv  Detail & Related papers  (2024-06-12T03:29:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.