ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments

• URL: http://arxiv.org/abs/2607.24964v1
• Date: Mon, 27 Jul 2026 18:13:28 GMT
• Title: ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments
• Abstract summary: Large language models are increasingly deployed for security-sensitive tasks such as vulnerability detection and code review.<n>Their reliance on natural-language context embedded in source code exposes a previously underexplored attack surface: adversarial comments.<n>We present ALIBI, an automated adaptive black-box attack framework that generates and iteratively refines adversarial comments.
• Score: 12.021872048536766
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly deployed for security-sensitive tasks such as vulnerability detection and code review. Their reliance on natural-language context embedded in source code exposes a previously underexplored attack surface: adversarial comments that can influence a detector's reasoning without changing program behavior. We study LLM-based vulnerability detectors against a new adversary: a coding agent that implements new functionality, deliberately introduces vulnerabilities, and strategically inserts adversarial source-code comments to evade detection. We present ALIBI, an automated adaptive black-box attack framework that generates and iteratively refines adversarial comments using detector reasoning and feedback. We transform real-world vulnerability-fixing commits into coding tasks and evaluate four representative LLM-based vulnerability detectors, ranging from specialized open-weight reasoning models to frontier multi-agent systems. All evaluated detectors are highly vulnerable: attack success rates exceed 90% across 125 real-world null-pointer dereference vulnerabilities, reaching 100% on one system. The framework also generalizes beyond this vulnerability class. Adversarial comments steering detector reasoning or fabricating external tool results prove most effective, while iterative refinement based on detector feedback further increases attack success. Finally, prompt-level defenses provide limited robustness against adaptive attacks, whereas architectural isolation and pre-detector comment sanitization substantially improve resilience. Our findings expose a fundamental attack surface in current LLM-based vulnerability detectors and motivate security-aware designs that carefully calibrate trust between natural-language context and program evidence.

Related papers

• PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections [51.8246149916068]
  Large Language Models (LLMs) are rapidly evolving into agentic systems that interact with external tools and environments.<n>We propose PI-Hunter, an automated agentic auditing framework for proactive vulnerability exposure in LLM agents.
  arXiv  Detail & Related papers  (2026-06-10T22:57:02Z)
• CIBER: A Comprehensive Benchmark for Security Evaluation of Code Interpreter Agents [27.35968236632966]
  LLM-based code interpreter agents are increasingly deployed in critical situations.<n>Existing benchmarks fail to capture the security risks arising from dynamic code execution, tool interactions, and multi-turn context.<n>We introduce CIBER, an automated benchmark that combines dynamic attack generation, isolated secure sandboxing, and state-aware evaluation.
  arXiv  Detail & Related papers  (2026-02-23T06:41:41Z)
• Semantics-Preserving Evasion of LLM Vulnerability Detectors [14.476903104601154]
  LLM-based vulnerability detectors are increasingly deployed in security-critical code review.<n>We evaluate detection-time integrity under a semantics-preserving threat model.<n>We introduce a metric of joint robustness across different attack methods/carriers.
  arXiv  Detail & Related papers  (2026-01-30T20:54:27Z)
• ICL-EVADER: Zero-Query Black-Box Evasion Attacks on In-Context Learning and Their Defenses [8.57098009274006]
  In-context learning (ICL) has become a powerful, data-efficient paradigm for text classification using large language models.<n>We introduce ICL-Evader, a novel black-box evasion attack framework that operates under a highly practical zero-query threat model.
  arXiv  Detail & Related papers  (2026-01-29T11:50:50Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search [58.8834056209347]
  Large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety guardrails to elicit harmful outputs.<n>We introduce the Correlated Knowledge Attack Agent (CKA-Agent), a dynamic framework that reframes jailbreaking as an adaptive, tree-structured exploration of the target model's knowledge base.
  arXiv  Detail & Related papers  (2025-12-01T07:05:23Z)
• Exploiting Web Search Tools of AI Agents for Data Exfiltration [0.46664938579243564]
  Large language models (LLMs) are now routinely used to execute complex tasks, from natural language processing to dynamic like web searches.<n>The usage of tool-calling and Retrieval Augmented Generation (RAG) allows LLMs to process and retrieve sensitive corporate data, amplifying both their functionality and vulnerability to abuse.<n>We analyze how susceptible current LLMs are to indirect prompt injection attacks, which parameters, including model size and manufacturer, shape their vulnerability, and which attack methods remain most effective.
  arXiv  Detail & Related papers  (2025-10-10T07:39:01Z)
• VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection [55.957275374847484]
  VulAgent is a multi-agent vulnerability detection framework based on hypothesis validation.<n>It implements a semantics-sensitive, multi-view detection pipeline, each aligned to a specific analysis perspective.<n>On average, VulAgent improves overall accuracy by 6.6%, increases the correct identification rate of vulnerable--fixed code pairs by up to 450%, and reduces the false positive rate by about 36%.
  arXiv  Detail & Related papers  (2025-09-15T02:25:38Z)
• BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attacks [58.959622170433725]
  BlindGuard is an unsupervised defense method that learns without requiring any attack-specific labels or prior knowledge of malicious behaviors.<n>We show that BlindGuard effectively detects diverse attack types (i.e., prompt injection, memory poisoning, and tool attack) across multi-agent systems.
  arXiv  Detail & Related papers  (2025-08-11T16:04:47Z)
• Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories [8.583591493627276]
  We introduce JitVul, a vulnerability detection benchmark linking each function to its vulnerability-introducing and fixing commits.<n>We show that ReAct Agents, leveraging thought-action-observation and interprocedural context, perform better than LLMs in distinguishing vulnerable from benign code.
  arXiv  Detail & Related papers  (2025-03-05T15:22:24Z)
• Reformulation is All You Need: Addressing Malicious Text Features in DNNs [53.45564571192014]
  We propose a unified and adaptive defense framework that is effective against both adversarial and backdoor attacks.<n>Our framework outperforms existing sample-oriented defense baselines across a diverse range of malicious textual features.
  arXiv  Detail & Related papers  (2025-02-02T03:39:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.