Bi-Directional Transformers vs. word2vec: Discovering Vulnerabilities in Lifted Compiled Code

• URL: http://arxiv.org/abs/2405.20611v3
• Date: Fri, 27 Sep 2024 13:29:00 GMT
• Title: Bi-Directional Transformers vs. word2vec: Discovering Vulnerabilities in Lifted Compiled Code
• Authors: Gary A. McCully, John D. Hastings, Shengjie Xu, Adam Fortier,
• Abstract summary: This research explores vulnerability detection using natural language processing (NLP) embedding techniques with word2vec, BERT, and RoBERTa. Long short-term memory (LSTM) neural networks were trained on embeddings from encoders created using approximately 48k LLVM functions from the Juliet dataset.
• Score: 4.956066467858057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Detecting vulnerabilities within compiled binaries is challenging due to lost high-level code structures and other factors such as architectural dependencies, compilers, and optimization options. To address these obstacles, this research explores vulnerability detection using natural language processing (NLP) embedding techniques with word2vec, BERT, and RoBERTa to learn semantics from intermediate representation (LLVM IR) code. Long short-term memory (LSTM) neural networks were trained on embeddings from encoders created using approximately 48k LLVM functions from the Juliet dataset. This study is pioneering in its comparison of word2vec models with multiple bidirectional transformers (BERT, RoBERTa) embeddings built using LLVM code to train neural networks to detect vulnerabilities in compiled binaries. Word2vec Skip-Gram models achieved 92% validation accuracy in detecting vulnerabilities, outperforming word2vec Continuous Bag of Words (CBOW), BERT, and RoBERTa. This suggests that complex contextual embeddings may not provide advantages over simpler word2vec models for this task when a limited number (e.g. 48K) of data samples are used to train the bidirectional transformer-based models. The comparative results provide novel insights into selecting optimal embeddings for learning compiler-independent semantic code representations to advance machine learning detection of vulnerabilities in compiled binaries.

Related papers

• SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognition [77.28814034644287]
  We propose SVTRv2, a CTC model that beats leading EDTRs in both accuracy and inference speed. SVTRv2 introduces novel upgrades to handle text irregularity and utilize linguistic context. We evaluate SVTRv2 in both standard and recent challenging benchmarks.
  arXiv  Detail & Related papers  (2024-11-24T14:21:35Z)
• Binary Code Similarity Detection via Graph Contrastive Learning on Intermediate Representations [52.34030226129628]
  Binary Code Similarity Detection (BCSD) plays a crucial role in numerous fields, including vulnerability detection, malware analysis, and code reuse identification. In this paper, we propose IRBinDiff, which mitigates compilation differences by leveraging LLVM-IR with higher-level semantic abstraction. Our extensive experiments, conducted under varied compilation settings, demonstrate that IRBinDiff outperforms other leading BCSD methods in both One-to-one comparison and One-to-many search scenarios.
  arXiv  Detail & Related papers  (2024-10-24T09:09:20Z)
• M2CVD: Enhancing Vulnerability Semantic through Multi-Model Collaboration for Code Vulnerability Detection [52.4455893010468]
  Large Language Models (LLMs) have strong capabilities in code comprehension, but fine-tuning costs and semantic alignment issues limit their project-specific optimization. Code models such CodeBERT are easy to fine-tune, but it is often difficult to learn vulnerability semantics from complex code languages. This paper introduces the Multi-Model Collaborative Vulnerability Detection approach (M2CVD) to improve the detection accuracy of code models.
  arXiv  Detail & Related papers  (2024-06-10T00:05:49Z)
• BinaryAI: Binary Software Composition Analysis via Intelligent Binary Source Code Matching [8.655595404611821]
  We introduce BinaryAI, a novel binary-to-source SCA technique with two-phase binary source code matching to capture both syntactic and semantic code features. Our experimental results demonstrate the superior performance of BinaryAI in terms of binary source code matching and the downstream SCA task.
  arXiv  Detail & Related papers  (2024-01-20T07:57:57Z)
• Feature Engineering-Based Detection of Buffer Overflow Vulnerability in Source Code Using Neural Networks [2.9266864570485827]
  vulnerability detection method based on neural network models that learn features extracted from source codes. We maintain the semantic and syntactic information using state of the art word embedding algorithms such as GloVe and fastText. We have proposed a neural network model that can overcome issues associated with traditional neural networks.
  arXiv  Detail & Related papers  (2023-06-01T01:44:49Z)
• Transformer-based approaches to Sentiment Detection [55.41644538483948]
  We examined the performance of four different types of state-of-the-art transformer models for text classification. The RoBERTa transformer model performs best on the test dataset with a score of 82.6% and is highly recommended for quality predictions.
  arXiv  Detail & Related papers  (2023-03-13T17:12:03Z)
• BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance [54.214426436283134]
  Deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications. We present a strong yet efficient binary neural network for KWS, namely BiFSMNv2, pushing it to the real-network accuracy performance. We highlight that benefiting from the compact architecture and optimized hardware kernel, BiFSMNv2 can achieve an impressive 25.1x speedup and 20.2x storage-saving on edge hardware.
  arXiv  Detail & Related papers  (2022-11-13T18:31:45Z)
• UniASM: Binary Code Similarity Detection without Fine-tuning [0.8271859911016718]
  We propose a novel transformer-based binary code embedding model named UniASM to learn representations of the binary functions. In the real-world task of known vulnerability search, UniASM outperforms all the current baselines.
  arXiv  Detail & Related papers  (2022-10-28T14:04:57Z)
• Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies [52.691032025163175]
  Existing Binary Neural Networks (BNNs) operate mainly on local convolutions with binarization function. We present new designs of binary neural modules, which enables leading binary neural modules by a large margin.
  arXiv  Detail & Related papers  (2022-09-03T11:51:04Z)
• Source Code Summarization with Structural Relative Position Guided Transformer [19.828300746504148]
  Source code summarization aims at generating concise and clear natural language descriptions for programming languages. Recent efforts focus on incorporating the syntax structure of code into neural networks such as Transformer. We propose a Structural Relative Position guided Transformer, named SCRIPT.
  arXiv  Detail & Related papers  (2022-02-14T07:34:33Z)
• Semantic-aware Binary Code Representation with BERT [27.908093567605484]
  A wide range of binary analysis applications, such as bug discovery, malware analysis and code clone detection, require recovery of contextual meanings on a binary code. Recently, binary analysis techniques based on machine learning have been proposed to automatically reconstruct the code representation of a binary. In this paper, we propose DeepSemantic utilizing BERT in producing the semantic-aware code representation of a binary code.
  arXiv  Detail & Related papers  (2021-06-10T03:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.