MalwarePT: A Binary-Level Foundation Model for Malware Analysis

• URL: http://arxiv.org/abs/2605.16455v1
• Date: Fri, 15 May 2026 05:31:59 GMT
• Title: MalwarePT: A Binary-Level Foundation Model for Malware Analysis
• Abstract summary: MalwarePT is a binary-level foundation model for malware analysis built on a ModernBERT-style encoder.<n>We train a byte-pair encoding (BPE) tokenizer on code-section bytes to compress frequent multi-byte patterns.
• Score: 18.754203608375704
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Automated malware analysis increasingly relies on machine learning, yet most existing methods remain task-specific and depend on handcrafted features or narrowly scoped models. Recent developments in binary-level foundation models suggest a path toward reusable program representations, but their application to malware analysis remains underexplored, and most still operate at byte-level tokenization, limiting their ability to capture multi-byte code patterns. In this work, we introduce MalwarePT, a binary-level foundation model for malware analysis built on a ModernBERT-style encoder and pretrained with masked language modeling on Windows PE code-section bytes. We study whether a single pretrained encoder can transfer across malware-analysis tasks at different granularities, and how tokenization design affects that transfer. We train a byte-pair encoding (BPE) tokenizer on code-section bytes to compress frequent multi-byte patterns within a fixed context budget. We evaluate MalwarePT on three downstream tasks spanning token-, function-, and document-level prediction: API call prediction, functionality classification, and malware (program) detection under temporal drift. Our evaluation demonstrates that pretraining yields substantial gains for API call prediction and functionality classification, and that increasing the BPE vocabulary beyond the byte-level baseline improves performance, with the strongest overall tradeoff at a vocabulary size of 1,024 tokens. In malware detection at FPR ~ 0.001, MalwarePT outperforms the neural network baselines, and is complementary to feature-engineering models that rely on PE structure. We also compare against existing binary foundation models and show that MalwarePT's design choices yield gains across all downstream tasks.

Related papers

• Large Byte Model: Teaching Language Models About Compiled Code [57.33411365104138]
  Large Language Models (LLMs) cannot process raw byte representations and answer questions about them.<n>We present the first byte-native LLM, based on a vocabulary expansion technique using a bespoke byte tokenizer.<n>Our findings indicate that providing domain knowledge during training is essential for this application.
  arXiv  Detail & Related papers  (2026-06-01T19:56:02Z)
• A Decompilation-Driven Framework for Malware Detection with Large Language Models [0.0]
  This paper evaluates the efficacy of Large Language Models (LLMs) in classifying executable code as either benign or malicious.<n>We introduce an automated pipeline that first decompiles Windows executable into a C code using Ghidra disassembler.<n>We demonstrate that a fine-tuned model, trained on curated malware and benign datasets, significantly outperforms its vanilla counterpart.
  arXiv  Detail & Related papers  (2026-01-14T00:00:26Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Semantic Preprocessing for LLM-based Malware Analysis [0.0]
  We propose a new preprocessing method which creates reports for Portable Executable files.<n>The purpose of this preprocessing is to gather a semantic representation of binary files, understandable by malware analysts.<n>Using this preprocessing to train a Large Language Model for Malware classification, we achieve a weighted-average F1-score of 0.94 on a complex dataset.
  arXiv  Detail & Related papers  (2025-06-13T13:39:00Z)
• Enhanced Consistency Bi-directional GAN(CBiGAN) for Malware Anomaly Detection [0.25163931116642785]
  This paper introduces the application of the CBiGAN in the domain of malware anomaly detection.<n>We utilize several datasets including both portable executable (PE) files as well as Object Linking and Embedding (OLE) files.<n>We then evaluated our model against a diverse set of both PE and OLE files, including self-collected malicious executables from 214 malware families.
  arXiv  Detail & Related papers  (2025-06-09T02:43:25Z)
• Zero Day Ransomware Detection with Pulse: Function Classification with Transformer Models and Assembly Language [1.870031206586792]
  Peekaboo, a Dynamic Binary Instrumentation tool defeats evasive malware to capture its genuine behavior. We propose Pulse, a novel framework for zero day ransomware detection with Transformer models and Assembly language.
  arXiv  Detail & Related papers  (2024-08-15T00:22:32Z)
• A Lean Transformer Model for Dynamic Malware Analysis and Detection [0.0]
  Malware is a fast-growing threat to the modern computing world and existing lines of defense are not efficient enough to address this issue. Previous works have shown some success leveraging Neural Networks and API calls sequences extracted from execution reports. In this paper, we design an emulation-Only model, based on the Transformers architecture, to detect malicious files.
  arXiv  Detail & Related papers  (2024-08-05T08:46:46Z)
• FoC: Figure out the Cryptographic Functions in Stripped Binaries with LLMs [51.898805184427545]
  We propose a novel framework called FoC to Figure out the Cryptographic functions in stripped binaries.<n>We first build a binary large language model (FoC-BinLLM) to summarize the semantics of cryptographic functions in natural language.<n>We then build a binary code similarity model (FoC-Sim) upon the FoC-BinLLM to create change-sensitive representations and use it to retrieve similar implementations of unknown cryptographic functions in a database.
  arXiv  Detail & Related papers  (2024-03-27T09:45:33Z)
• Code Representation Learning At Scale [75.04686476303436]
  We fuel code representation learning with a vast amount of code data via a two-stage pretraining scheme. We first train the encoders via a mix that leverages both randomness in masking language modeling and the structure aspect of programming language. We then enhance the representations via contrastive learning with hard negative and hard positive constructed in an unsupervised manner.
  arXiv  Detail & Related papers  (2024-02-02T22:19:15Z)
• DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness [58.23214712926585]
  We develop a certified defense, DRSM (De-Randomized Smoothed MalConv), by redesigning the de-randomized smoothing technique for the domain of malware detection. Specifically, we propose a window ablation scheme to provably limit the impact of adversarial bytes while maximally preserving local structures of the executables. We are the first to offer certified robustness in the realm of static detection of malware executables.
  arXiv  Detail & Related papers  (2023-03-20T17:25:22Z)
• Software Vulnerability Detection via Deep Learning over Disaggregated Code Graph Representation [57.92972327649165]
  This work explores a deep learning approach to automatically learn the insecure patterns from code corpora. Because code naturally admits graph structures with parsing, we develop a novel graph neural network (GNN) to exploit both the semantic context and structural regularity of a program.
  arXiv  Detail & Related papers  (2021-09-07T21:24:36Z)
• Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection [67.53296659361598]
  adversarial EXEmples can bypass machine learning-based detection by perturbing relatively few input bytes. We develop a unifying framework that does not only encompass and generalize previous attacks against machine-learning models, but also includes three novel attacks. These attacks, named Full DOS, Extend and Shift, inject the adversarial payload by respectively manipulating the DOS header, extending it, and shifting the content of the first section.
  arXiv  Detail & Related papers  (2020-08-17T07:16:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.