Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscation

• URL: http://arxiv.org/abs/2412.16135v3
• Date: Wed, 29 Jan 2025 13:52:31 GMT
• Title: Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscation
• Authors: Seyedreza Mohseni, Seyedali Mohammadi, Deepa Tilwani, Yash Saxena, Gerald Ketu Ndawula, Sriram Vema, Edward Raff, Manas Gaur,
• Abstract summary: Malware authors often employ code obfuscations to make their malware harder to detect.<n>Existing tools for generating obfuscated code often require access to the original source code.<n>Can Large Language Models potentially generate a new obfuscated assembly code?<n>If so, this poses a risk to anti-virus engines and potentially increases the flexibility of attackers to create new obfuscation patterns.
• Score: 36.12009987721901
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Malware authors often employ code obfuscations to make their malware harder to detect. Existing tools for generating obfuscated code often require access to the original source code (e.g., C++ or Java), and adding new obfuscations is a non-trivial, labor-intensive process. In this study, we ask the following question: Can Large Language Models (LLMs) potentially generate a new obfuscated assembly code? If so, this poses a risk to anti-virus engines and potentially increases the flexibility of attackers to create new obfuscation patterns. We answer this in the affirmative by developing the MetamorphASM benchmark comprising MetamorphASM Dataset (MAD) along with three code obfuscation techniques: dead code, register substitution, and control flow change. The MetamorphASM systematically evaluates the ability of LLMs to generate and analyze obfuscated code using MAD, which contains 328,200 obfuscated assembly code samples. We release this dataset and analyze the success rate of various LLMs (e.g., GPT-3.5/4, GPT-4o-mini, Starcoder, CodeGemma, CodeLlama, CodeT5, and LLaMA 3.1) in generating obfuscated assembly code. The evaluation was performed using established information-theoretic metrics and manual human review to ensure correctness and provide the foundation for researchers to study and develop remediations to this risk.

Related papers

• Simplicity by Obfuscation: Evaluating LLM-Driven Code Transformation with Semantic Elasticity [4.458584890504334]
  Code obfuscation aims to prevent reverse engineering and intellectual property theft. The recent development of large language models paves the way for practical applications in different domains. This work performs an empirical study on the ability of LLMs to obfuscate Python source code.
  arXiv  Detail & Related papers  (2025-04-18T18:29:23Z)
• The Code Barrier: What LLMs Actually Understand? [7.407441962359689]
  This research uses code obfuscation as a structured testing framework to evaluate semantic understanding capabilities of language models. Findings show a statistically significant performance decline as obfuscation complexity increases. This research introduces a new evaluation approach for assessing code comprehension in language models.
  arXiv  Detail & Related papers  (2025-04-14T14:11:26Z)
• ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding [60.37988508851391]
  Language models (LMs) have become a staple of the code-writing toolbox. Research exploring modifications to Code-LMs' pre-training objectives, geared towards improving data efficiency and better disentangling between syntax and semantics, has been noticeably sparse. In this work, we examine grounding on obfuscated code as a means of helping Code-LMs look beyond the surface-form syntax and enhance their pre-training sample efficiency.
  arXiv  Detail & Related papers  (2025-03-27T23:08:53Z)
• Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar [15.421030528350212]
  We build a code-obfuscation based benchmark OBFUSEVAL to evaluate large language models.<n>We use three-level strategy to obfuscate descriptions, code and context dependencies.<n>The results show that after obfuscation, the average decrease ratio of test pass rate can up to 62.5%.
  arXiv  Detail & Related papers  (2024-12-11T05:31:39Z)
• CodeCipher: Learning to Obfuscate Source Code Against LLMs [5.872773591957006]
  We propose CodeCipher, a novel method that perturbs privacy from code while preserving the original response from LLMs. CodeCipher transforms the LLM's embedding matrix so that each row corresponds to a different word in the original matrix, forming a token-to-token confusion mapping for obfuscating source code. Results show that our model successfully confuses the privacy in source code while preserving the original LLM's performance.
  arXiv  Detail & Related papers  (2024-10-08T08:28:54Z)
• Artificial-Intelligence Generated Code Considered Harmful: A Road Map for Secure and High-Quality Code Generation [2.793781561647737]
  We compared the security and quality of human-written code with that of LLM-generated code. We found that LLM can generate incorrect code that fails to implement the required functionality. Flukeing has revealed that LLM-generated code is more prone to hangs and crashes than human-written code.
  arXiv  Detail & Related papers  (2024-09-27T23:41:51Z)
• HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Data [60.75578581719921]
  Large language models (LLMs) have shown great potential for automatic code generation. Recent studies highlight that many LLM-generated code contains serious security vulnerabilities. We introduce HexaCoder, a novel approach to enhance the ability of LLMs to generate secure codes.
  arXiv  Detail & Related papers  (2024-09-10T12:01:43Z)
• VersiCode: Towards Version-controllable Code Generation [58.82709231906735]
  Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development. We propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM) We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge.
  arXiv  Detail & Related papers  (2024-06-11T16:15:06Z)
• Bugs in Large Language Models Generated Code: An Empirical Study [12.625305075672456]
  Large Language Models (LLMs) for code have gained significant attention recently. Similar to human-written code, LLM-generated code is prone to bugs. This paper examines a sample of 333 bugs collected from code generated using three leading LLMs.
  arXiv  Detail & Related papers  (2024-03-13T20:12:01Z)
• CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion [117.178835165855]
  This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs. Our studies reveal a new and universal safety vulnerability of these models against code input. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization.
  arXiv  Detail & Related papers  (2024-03-12T17:55:38Z)
• Assured LLM-Based Software Engineering [51.003878077888686]
  This paper is an outline of the content of the keynote by Mark Harman at the International Workshop on Interpretability, Robustness, and Benchmarking in Neural Software Engineering, Monday 15th April 2024, Lisbon, Portugal.
  arXiv  Detail & Related papers  (2024-02-06T20:38:46Z)
• Chain of Code: Reasoning with a Language Model-Augmented Code Emulator [115.16975276693267]
  We propose Chain of Code, a simple yet surprisingly effective extension that improves LM code-driven reasoning. The key idea is to encourage LMs to format semantic sub-tasks in a program as flexible pseudocode that the interpreter can explicitly catch.
  arXiv  Detail & Related papers  (2023-12-07T17:51:43Z)
• Zero-Shot Detection of Machine-Generated Codes [83.0342513054389]
  This work proposes a training-free approach for the detection of LLMs-generated codes. We find that existing training-based or zero-shot text detectors are ineffective in detecting code. Our method exhibits robustness against revision attacks and generalizes well to Java codes.
  arXiv  Detail & Related papers  (2023-10-08T10:08:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.