Practical Attacks against Black-box Code Completion Engines

• URL: http://arxiv.org/abs/2408.02509v1
• Date: Mon, 5 Aug 2024 14:31:26 GMT
• Title: Practical Attacks against Black-box Code Completion Engines
• Authors: Slobodan Jenko, Jingxuan He, Niels Mündler, Mark Vero, Martin Vechev,
• Abstract summary: We present INSEC, a novel attack that directs code completion engines towards generating vulnerable code. In line with most commercial completion engines, such as GitHub Copilot, INSEC assumes only black-box query access to the targeted engine. Our attack works by inserting a malicious attack string as a short comment in the completion input.
• Score: 5.633172380505533
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern code completion engines, powered by large language models, have demonstrated impressive capabilities to generate functionally correct code based on surrounding context. As these tools are extensively used by millions of developers, it is crucial to investigate their security implications. In this work, we present INSEC, a novel attack that directs code completion engines towards generating vulnerable code. In line with most commercial completion engines, such as GitHub Copilot, INSEC assumes only black-box query access to the targeted engine, without requiring any knowledge of the engine's internals. Our attack works by inserting a malicious attack string as a short comment in the completion input. To derive the attack string, we design a series of specialized initialization schemes and an optimization procedure for further refinement. We demonstrate the strength of INSEC not only on state-of-the-art open-source models but also on black-box commercial services such as the OpenAI API and GitHub Copilot. On a comprehensive set of security-critical test cases covering 16 CWEs across 5 programming languages, INSEC significantly increases the likelihood of the considered completion engines in generating unsafe code by >50% in absolute, while maintaining the ability in producing functionally correct code. At the same time, our attack has low resource requirements, and can be developed for a cost of well under ten USD on commodity hardware.

Related papers

• Wolves in the Repository: A Software Engineering Analysis of the XZ Utils Supply Chain Attack [0.8517406772939294]
  The digital economy runs on Open Source Software (OSS), with an estimated 90% of modern applications containing open-source components. This paper examines a sophisticated attack on the XZUtils project (-2024-3094), where attackers exploited not just code, but the entire open-source development process. Our analysis reveals a new breed of supply chain attack that manipulates software engineering practices themselves.
  arXiv  Detail & Related papers  (2025-04-24T12:06:11Z)
• RedCode: Risky Code Execution and Generation Benchmark for Code Agents [50.81206098588923]
  RedCode is a benchmark for risky code execution and generation. RedCode-Exec provides challenging prompts that could lead to risky code execution. RedCode-Gen provides 160 prompts with function signatures and docstrings as input to assess whether code agents will follow instructions.
  arXiv  Detail & Related papers  (2024-11-12T13:30:06Z)
• An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detection [17.948513691133037]
  We introduce CodeBreaker, a pioneering LLM-assisted backdoor attack framework on code completion models. By integrating malicious payloads directly into the source code with minimal transformation, CodeBreaker challenges current security measures.
  arXiv  Detail & Related papers  (2024-06-10T22:10:05Z)
• FV8: A Forced Execution JavaScript Engine for Detecting Evasive Techniques [53.288368877654705]
  FV8 is a modified V8 JavaScript engine designed to identify evasion techniques in JavaScript code. It selectively enforces code execution on APIs that conditionally inject dynamic code. It identifies 1,443 npm packages and 164 (82%) extensions containing at least one type of evasion.
  arXiv  Detail & Related papers  (2024-05-21T19:54:19Z)
• Enhancing Security of AI-Based Code Synthesis with GitHub Copilot via Cheap and Efficient Prompt-Engineering [1.7702475609045947]
  One of the reasons developers and companies avoid harnessing their full potential is the questionable security of the generated code. This paper first reviews the current state-of-the-art and identifies areas for improvement on this issue. We propose a systematic approach based on prompt-altering methods to achieve better code security of AI-based code generators such as GitHub Copilot.
  arXiv  Detail & Related papers  (2024-03-19T12:13:33Z)
• CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion [117.178835165855]
  This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs. Our studies reveal a new and universal safety vulnerability of these models against code input. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization.
  arXiv  Detail & Related papers  (2024-03-12T17:55:38Z)
• LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Reward [3.729516018513228]
  We introduce a multipurpose code vulnerability analysis system textttSecRepair, powered by a large language model, CodeGen2. Inspired by how humans fix code issues, we propose an instruction-based dataset suitable for vulnerability analysis with LLMs. We identify zero-day and N-day vulnerabilities in 6 Open Source IoT Operating Systems on GitHub.
  arXiv  Detail & Related papers  (2024-01-07T02:46:39Z)
• A LLM Assisted Exploitation of AI-Guardian [57.572998144258705]
  We evaluate the robustness of AI-Guardian, a recent defense to adversarial examples published at IEEE S&P 2023. We write none of the code to attack this model, and instead prompt GPT-4 to implement all attack algorithms following our instructions and guidance. This process was surprisingly effective and efficient, with the language model at times producing code from ambiguous instructions faster than the author of this paper could have done.
  arXiv  Detail & Related papers  (2023-07-20T17:33:25Z)
• Backdooring Neural Code Search [20.88291603306741]
  An adversary can inject a backdoor in neural code search models. In this paper, we demonstrate such attacks are feasible and can be quite stealthy. Our attack BADCODE features a special trigger generation and injection procedure, making the attack more effective and stealthy.
  arXiv  Detail & Related papers  (2023-05-27T16:00:50Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)
• CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models [58.27254444280376]
  Large language models (LLMs) for automatic code generation have achieved breakthroughs in several programming tasks. Training data for these models is usually collected from the Internet (e.g., from open-source repositories) and is likely to contain faults and security vulnerabilities. This unsanitized training data can cause the language models to learn these vulnerabilities and propagate them during the code generation procedure.
  arXiv  Detail & Related papers  (2023-02-08T11:54:07Z)
• Predicting Vulnerability In Large Codebases With Deep Code Representation [6.357681017646283]
  Software engineers write code for various modules, quite often, various types of errors get introduced. Same or similar issues/bugs, which were fixed in the past (although in different modules), tend to get introduced in production code again. We developed a novel AI-based system which uses the deep representation of Abstract Syntax Tree (AST) created from the source code and also the active feedback loop.
  arXiv  Detail & Related papers  (2020-04-24T13:18:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.