Large Byte Model: Teaching Language Models About Compiled Code

• URL: http://arxiv.org/abs/2606.02834v1
• Date: Mon, 01 Jun 2026 19:56:02 GMT
• Title: Large Byte Model: Teaching Language Models About Compiled Code
• Abstract summary: Large Language Models (LLMs) cannot process raw byte representations and answer questions about them.<n>We present the first byte-native LLM, based on a vocabulary expansion technique using a bespoke byte tokenizer.<n>Our findings indicate that providing domain knowledge during training is essential for this application.
• Score: 57.33411365104138
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Malware analysis starts with the raw bytes of an executable program, and tools to "lift" these to higher-level representations, such as assembly, are expensive and subject to error. Large Language Models (LLMs) cannot process raw byte representations and answer questions about them. To this end, we present the first byte-native LLM. Based on a vocabulary expansion technique using a bespoke byte tokenizer, such a model is capable of responding to complex questions about malware binaries, with accuracies ranging from 69% for malware family classification to 98% for architecture classification. Our findings indicate that providing domain knowledge during training is essential for this application -- off-the-shelf models lack both accuracy and insight. We've deployed this emerging solution to a limited number of analysts to gather feedback for further improvements.

Related papers

• Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphs [79.8277120258141]
  Knowledge Graphs (KGs) are widely used to mitigate the limitations of Large Language Models (LLMs)<n>Existing LLM-KG integration frameworks rely on predefined operators to retrieve factual knowledge from KGs and inject it into prompts for answer generation.<n>We propose Code-on-Graph (CoG) as a programmatic reasoning framework for LLM-KG integration.
  arXiv  Detail & Related papers  (2026-06-02T14:22:29Z)
• MalwarePT: A Binary-Level Foundation Model for Malware Analysis [18.754203608375704]
  MalwarePT is a binary-level foundation model for malware analysis built on a ModernBERT-style encoder.<n>We train a byte-pair encoding (BPE) tokenizer on code-section bytes to compress frequent multi-byte patterns.
  arXiv  Detail & Related papers  (2026-05-15T05:31:59Z)
• Semantic Preprocessing for LLM-based Malware Analysis [0.0]
  We propose a new preprocessing method which creates reports for Portable Executable files.<n>The purpose of this preprocessing is to gather a semantic representation of binary files, understandable by malware analysts.<n>Using this preprocessing to train a Large Language Model for Malware classification, we achieve a weighted-average F1-score of 0.94 on a complex dataset.
  arXiv  Detail & Related papers  (2025-06-13T13:39:00Z)
• The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs [45.08958917457921]
  Large language models (LLMs) still struggle across tasks outside of high-resource languages.<n>In this work, we investigate cross-lingual transfer to lower-resource languages where task-specific post-training data is scarce.
  arXiv  Detail & Related papers  (2025-05-23T20:28:31Z)
• Learnware of Language Models: Specialized Small Language Models Can Do Big [50.285859986475394]
  This paper presents a preliminary attempt to apply the learnware paradigm to language models.<n>We simulated a learnware system comprising approximately 100 learnwares of specialized SLMs with 8B parameters.<n>By selecting one suitable learnware for each task-specific inference, the system outperforms the base SLMs on all benchmarks.
  arXiv  Detail & Related papers  (2025-05-19T17:54:35Z)
• Scalable APT Malware Classification via Parallel Feature Extraction and GPU-Accelerated Learning [0.3277163122167433]
  This paper presents a framework for mapping malicious executables to known Persistent Advanced Threat (APT) groups.<n>The main feature of this analysis is the assembly-level instructions present in executables which are also known as opcodes.<n>Traditional and deep learning models are applied to create models capable of classifying malware samples.
  arXiv  Detail & Related papers  (2025-04-22T00:05:05Z)
• Understanding Information Storage and Transfer in Multi-modal Large Language Models [51.20840103605018]
  We study how Multi-modal Large Language Models process information in a factual visual question answering task. Key findings show that these MLLMs rely on self-attention blocks in much earlier layers for information storage. We introduce MultEdit, a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs.
  arXiv  Detail & Related papers  (2024-06-06T16:35:36Z)
• Mixture-of-Instructions: Aligning Large Language Models via Mixture Prompting [7.103987978402038]
  We introduce a novel technique termed Mixture-of-Instructions (MoI)<n>MoI employs a strategy of instruction packing combined with diverse system prompts to boost the alignment efficiency of language models.<n>Our methodology was applied to the open-source Qwen-7B-chat model, culminating in the development of Qwen-SFT-MoI.
  arXiv  Detail & Related papers  (2024-04-29T03:58:12Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.