Early Comparative Evaluation of Transformer Models for Multilingual Software Vulnerability Detection

• URL: http://arxiv.org/abs/2606.10925v1
• Date: Tue, 09 Jun 2026 14:33:34 GMT
• Title: Early Comparative Evaluation of Transformer Models for Multilingual Software Vulnerability Detection
• Abstract summary: This paper presents an early comparative evaluation of BERT, RoBERTa, and CodeBERT for binary vulnerability detection across HTML, Python, JavaScript, and PHP.<n>Results show clear performance differences across languages, indicating that multilingual vulnerability detection requires more language-aware and robust transformer-based modelling strategies.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software vulnerability detection is increasingly important as modern applications combine multiple programming languages. This paper presents an early comparative evaluation of BERT, RoBERTa, and CodeBERT for binary vulnerability detection across HTML, Python, JavaScript, and PHP using the CVEFixes dataset and language-wise three-fold stratified cross-validation. The results show clear performance differences across languages, indicating that multilingual vulnerability detection requires more language-aware and robust transformer-based modelling strategies.

Related papers

• Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation [49.2073409243885]
  Large language models (LLMs) excel at generating English counterfactuals and demonstrate multilingual proficiency.<n>We conduct automatic evaluations on both directly generated counterfactuals in the target languages and those derived via English translation across six languages.<n>We identify and categorize four main types of errors that consistently appear in the generated counterfactuals across languages.
  arXiv  Detail & Related papers  (2026-01-01T08:53:49Z)
• Asm2SrcEval: Evaluating Large Language Models for Assembly-to-Source Code Translation [4.45354703148321]
  Assembly-to-source code translation is a critical task in reverse engineering, cybersecurity, and software maintenance.<n>We present the first comprehensive evaluation of five state-of-the-art large language models on assembly-to-source translation.
  arXiv  Detail & Related papers  (2025-11-28T12:40:30Z)
• MulVuln: Enhancing Pre-trained LMs with Shared and Language-Specific Knowledge for Multilingual Vulnerability Detection [22.197550174544627]
  MULVULN is a novel multilingual vulnerability detection approach that learns from source code across multiple languages.<n>It achieves more robust and effective detection of vulnerabilities in real-world multilingual software systems.<n> Notably, MULVULN achieves substantially higher F1-score, with improvements ranging from 1.45% to 23.59% compared to the baseline methods.
  arXiv  Detail & Related papers  (2025-10-05T23:33:26Z)
• Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach [51.95266411355865]
  Autoregressive language models are vulnerable to orthographic attacks.<n>This vulnerability stems from the out-of-vocabulary issue inherent in subword tokenizers and their embeddings.<n>We propose a pixel-based generative language model that replaces the text-based embeddings with pixel-based representations by rendering words as individual images.
  arXiv  Detail & Related papers  (2025-08-28T20:48:38Z)
• Enhancing Large Language Models with Faster Code Preprocessing for Vulnerability Detection [0.0]
  We build on the existing SCoPE framework and introduce SCoPE2, an enhanced version with improved performance.<n>Our results show a 97.3% reduction in processing time with SCoPE2, along with an improved F1-score for the Large Language Model (LLM) for vulnerability detection.
  arXiv  Detail & Related papers  (2025-05-08T19:00:11Z)
• Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training [58.696660064190475]
  We find that the existence of code-switching, alternating between different languages within a context, is key to multilingual capabilities.<n>To better explore the power of code-switching for language alignment during pre-training, we investigate the strategy of synthetic code-switching.
  arXiv  Detail & Related papers  (2025-04-02T15:09:58Z)
• Vulnerability Detection in Popular Programming Languages with Language Models [2.048226951354646]
  This paper investigates the effectiveness of Language Models (LMs) for vulnerability detection in JavaScript, Java, Python, PHP, and Go.<n>We find that JavaScript exhibits the best performance, with considerably better and more practical detection capabilities compared to C/C++.
  arXiv  Detail & Related papers  (2024-12-20T13:57:27Z)
• Development and Benchmarking of Multilingual Code Clone Detector [2.253851493296371]
  multilingual code clone detectors make it easier to add new language support by providing syntax information of the target language only. We propose a multilingual code block extraction method based on ANTLR generation and implement a multilingual code clone detector (MSCCD) Compared to ten state-of-the-art detectors, MSCCD performs at an average level while it also supports a significantly larger number of languages.
  arXiv  Detail & Related papers  (2024-09-10T03:08:33Z)
• AdaCCD: Adaptive Semantic Contrasts Discovery Based Cross Lingual Adaptation for Code Clone Detection [69.79627042058048]
  AdaCCD is a novel cross-lingual adaptation method that can detect cloned codes in a new language without annotations in that language. We evaluate the cross-lingual adaptation results of AdaCCD by constructing a multilingual code clone detection benchmark consisting of 5 programming languages.
  arXiv  Detail & Related papers  (2023-11-13T12:20:48Z)
• XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding [73.24847320536813]
  This study explores distilling visual information from pretrained multimodal transformers to pretrained language encoders. Our framework is inspired by cross-modal encoders' success in visual-language tasks while we alter the learning objective to cater to the language-heavy characteristics of NLU.
  arXiv  Detail & Related papers  (2022-04-15T03:44:00Z)
• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective [84.78604733927887]
  Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models.
  arXiv  Detail & Related papers  (2020-10-05T20:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.