EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers

• URL: http://arxiv.org/abs/2506.05074v1
• Date: Thu, 05 Jun 2025 14:20:36 GMT
• Title: EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers
• Authors: Robert J. Joyce, Gideon Miller, Phil Roth, Richard Zak, Elliott Zaresky-Williams, Hyrum Anderson, Edward Raff, James Holt,
• Abstract summary: We present EMBER2024, a new dataset that enables holistic evaluation of malware classifiers.<n>Our dataset supports the training and evaluation of machine learning models on seven malware classification tasks.<n> EMBER2024 is the first to include a collection of malicious files that initially went undetected by a set of antivirus products.
• Score: 34.77788258445852
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A lack of accessible data has historically restricted malware analysis research, and practitioners have relied heavily on datasets provided by industry sources to advance. Existing public datasets are limited by narrow scope - most include files targeting a single platform, have labels supporting just one type of malware classification task, and make no effort to capture the evasive files that make malware detection difficult in practice. We present EMBER2024, a new dataset that enables holistic evaluation of malware classifiers. Created in collaboration with the authors of EMBER2017 and EMBER2018, the EMBER2024 dataset includes hashes, metadata, feature vectors, and labels for more than 3.2 million files from six file formats. Our dataset supports the training and evaluation of machine learning models on seven malware classification tasks, including malware detection, malware family classification, and malware behavior identification. EMBER2024 is the first to include a collection of malicious files that initially went undetected by a set of antivirus products, creating a "challenge" set to assess classifier performance against evasive malware. This work also introduces EMBER feature version 3, with added support for several new feature types. We are releasing the EMBER2024 dataset to promote reproducibility and empower researchers in the pursuit of new malware research topics.

Related papers

• Malware families discovery via Open-Set Recognition on Android manifest permissions [15.838751258859004]
  Classifying malware programs into their respective families is essential for building effective defenses against cyber threats.<n>We present a malware classification system that, on top of classifying known malware, detects new ones.<n>Our solution turns out to be very practical, as it can be seamlessly employed in a standard classification workflow.
  arXiv  Detail & Related papers  (2025-05-19T06:19:54Z)
• Multi-label Classification for Android Malware Based on Active Learning [7.599125552187342]
  We propose MLCDroid, an ML-based multi-label classification approach that can directly indicate the existence of pre-defined malicious behaviors. We compare the results of 70 algorithm combinations to evaluate the effectiveness (best at 73.3%). This is the first multi-label Android malware classification approach intending to provide more information on fine-grained malicious behaviors.
  arXiv  Detail & Related papers  (2024-10-09T01:09:24Z)
• MalDICT: Benchmark Datasets on Malware Behaviors, Platforms, Exploitation, and Packers [44.700094741798445]
  Existing research on malware classification focuses almost exclusively on two tasks: distinguishing between malicious and benign files and classifying malware by family. We have identified four tasks which are under-represented in prior work: classification by behaviors that malware exhibit, platforms that malware run on, vulnerabilities that malware exploit, and packers that malware are packed with. We are releasing benchmark datasets for each of these four classification tasks, tagged using ClarAVy and comprising nearly 5.5 million malicious files in total.
  arXiv  Detail & Related papers  (2023-10-18T04:36:26Z)
• EMBERSim: A Large-Scale Databank for Boosting Similarity Search in Malware Analysis [48.5877840394508]
  In recent years there has been a shift from quantifications-based malware detection towards machine learning. We propose to address the deficiencies in the space of similarity research on binary files, starting from EMBER. We enhance EMBER with similarity information as well as malware class tags, to enable further research in the similarity space.
  arXiv  Detail & Related papers  (2023-10-03T06:58:45Z)
• DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness [58.23214712926585]
  We develop a certified defense, DRSM (De-Randomized Smoothed MalConv), by redesigning the de-randomized smoothing technique for the domain of malware detection. Specifically, we propose a window ablation scheme to provably limit the impact of adversarial bytes while maximally preserving local structures of the executables. We are the first to offer certified robustness in the realm of static detection of malware executables.
  arXiv  Detail & Related papers  (2023-03-20T17:25:22Z)
• Behavioural Reports of Multi-Stage Malware [3.64414368529873]
  This dataset provides API call sequences for thousands of malware samples executed in Windows 10 virtual machines. A tutorial on how to create and expand this dataset is provided along with a benchmark demonstrating how to use this dataset to classify malware.
  arXiv  Detail & Related papers  (2023-01-30T11:51:02Z)
• New Datasets for Dynamic Malware Classification [0.0]
  We introduce two new, updated datasets of malicious software, VirusSamples and VirusShare. This paper analyzes multi-class malware classification performance of the balanced and imbalanced version of these two datasets. Results show that Support Vector Machine, achieves the highest score of 94% in the imbalanced VirusSample dataset. XGBoost, one of the most common gradient boosting-based models, achieves the highest score of 90% and 80%.in both versions of the VirusShare dataset.
  arXiv  Detail & Related papers  (2021-11-30T08:31:16Z)
• Being Single Has Benefits. Instance Poisoning to Deceive Malware Classifiers [47.828297621738265]
  We show how an attacker can launch a sophisticated and efficient poisoning attack targeting the dataset used to train a malware classifier. As opposed to other poisoning attacks in the malware detection domain, our attack does not focus on malware families but rather on specific malware instances that contain an implanted trigger. We propose a comprehensive detection approach that could serve as a future sophisticated defense against this newly discovered severe threat.
  arXiv  Detail & Related papers  (2020-10-30T15:27:44Z)
• Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection [67.53296659361598]
  adversarial EXEmples can bypass machine learning-based detection by perturbing relatively few input bytes. We develop a unifying framework that does not only encompass and generalize previous attacks against machine-learning models, but also includes three novel attacks. These attacks, named Full DOS, Extend and Shift, inject the adversarial payload by respectively manipulating the DOS header, extending it, and shifting the content of the first section.
  arXiv  Detail & Related papers  (2020-08-17T07:16:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.