XuanJia: A Comprehensive Virtualization-Based Code Obfuscator for Binary Protection

• URL: http://arxiv.org/abs/2601.10261v1
• Date: Thu, 15 Jan 2026 10:33:47 GMT
• Title: XuanJia: A Comprehensive Virtualization-Based Code Obfuscator for Binary Protection
• Abstract summary: XuanJia is a comprehensive VM-based binary obfuscation framework.<n>It provides end-to-end protection for both executable code and exception-handling semantics.
• Score: 13.391445401824042
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Virtualization-based binary obfuscation is widely adopted to protect software intellectual property, yet existing approaches leave exception-handling (EH) metadata unprotected to preserve ABI compatibility. This exposed metadata leaks rich structural information, such as stack layouts, control-flow boundaries, and object lifetimes, which can be exploited to facilitate reverse engineering. In this paper, we present XuanJia, a comprehensive VM-based binary obfuscation framework that provides end-to-end protection for both executable code and exception-handling semantics. At the core of XuanJia is ABI-Compliant EH Shadowing, a novel exception-aware protection mechanism that preserves compatibility with unmodified operating system runtimes while eliminating static EH metadata leakage. XuanJia replaces native EH metadata with ABI-compliant shadow unwind information to satisfy OS-driven unwinding, and securely redirects exception handling into a protected virtual machine where the genuine EH semantics are decrypted, reversed, and replayed using obfuscated code. We implement XuanJia from scratch, supporting 385 x86 instruction encodings and 155 VM handler templates, and design it as an extensible research testbed. We evaluate XuanJia across correctness, resilience, and performance dimensions. Our results show that XuanJia preserves semantic equivalence under extensive dynamic and symbolic testing, effectively disrupts automated reverse-engineering tools such as IDA Pro, and incurs negligible space overhead and modest runtime overhead. These results demonstrate that XuanJia achieves strong protection of exception-handling logic without sacrificing correctness or practicality.

Related papers

• A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks [53.12252890906191]
  Large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose a self-evolving test-time defense built around a persistent, cross-interaction rule memory.
  arXiv  Detail & Related papers  (2026-08-26T16:52:06Z)
• Automation-Exploit: A Multi-Agent LLM Framework for Adaptive Offensive Security with Digital Twin-Based Risk-Mitigated Exploitation [0.05729426778193397]
  We introduce Automation-Exploit, a fully autonomous Multi-Agent System (MAS) framework for adaptive offensive security.<n>It bridges the abstraction gap between reconnaissance and exploitation by autonomously exfiltrating executables.<n>It employs a conditional isomorphic validation for high-risk memory-corruption flaws.
  arXiv  Detail & Related papers  (2026-04-24T10:38:40Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Pushan: Trace-Free Deobfuscation of Virtualization-Obfuscated Binaries [18.483613577646974]
  Existing automated deobfuscation techniques suffer from three major drawbacks.<n>They only work on execution traces, which prevents them from recovering all logic in an obfuscated binary.<n> PUSHAN is trace-free and avoids path-constraint accumulation.<n>It is the first approach that also decompiles the protected code into high-quality C pseudocode to enable effective analysis.
  arXiv  Detail & Related papers  (2026-03-18T23:37:39Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• ShieldedCode: Learning Robust Representations for Virtual Machine Protected Code [9.803721207708941]
  ShieldedCode is the first protection-aware framework that learns robust representations of VMP-protected code.<n>Our method achieves 26.95% Pass@1 on L0 VM code generation compared to 22.58% for GPT-4o, and improves binary similarity detection Recall@1 by 10% over state of art methods like jTrans.
  arXiv  Detail & Related papers  (2026-01-28T15:07:08Z)
• Exposing Hidden Interfaces: LLM-Guided Type Inference for Reverse Engineering macOS Private Frameworks [7.370020034607549]
  MOTIF is an agentic framework that integrates tool-augmented analysis with a large language model specialized for Objective-C type inference.<n>By transforming binaries into analyzable interfaces, MOTIF establishes a scalable foundation for systematic auditing of runtime internals.
  arXiv  Detail & Related papers  (2026-01-04T21:44:55Z)
• RAID: Refusal-Aware and Integrated Decoding for Jailbreaking LLMs [17.313975711973374]
  RAID (Refusal-Aware and Integrated Decoding) is a framework that crafts adversarial suffixes that induce restricted content while preserving fluency.<n>We show that RAID achieves higher attack success rates with fewer queries and lower computational cost than recent white-box and black-box baselines.
  arXiv  Detail & Related papers  (2025-10-14T19:33:09Z)
• Obelix: Mitigating Side-Channels Through Dynamic Obfuscation [8.043868787597109]
  We take a holistic view at the vulnerabilities of TEEs and design a tool named Obelix.<n>Obelix is the first to protect both code and data against a wide range of TEE attacks.
  arXiv  Detail & Related papers  (2025-09-23T12:32:51Z)
• RX-INT: A Kernel Engine for Real-Time Detection and Analysis of In-Memory Threats [0.0]
  We present RX-INT, a kernel-assisted system featuring an architecture that provides resilience against TOCTOU attacks.<n> RX-INT introduces a detection engine that combines a real-time thread creation monitor with a stateful Virtual Address Descriptor (VAD) scanner.<n>In our evaluation, RX-INT successfully detected a manually mapped region that was not identified by PE-sieve.
  arXiv  Detail & Related papers  (2025-08-05T19:43:25Z)
• The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMs [39.85609149662187]
  We present DIJA, the first systematic study and jailbreak attack framework that exploits unique safety weaknesses of dLLMs.<n>Our proposed DIJA constructs adversarial interleaved mask-text prompts that exploit the text generation mechanisms of dLLMs.<n>Our findings underscore the urgent need for rethinking safety alignment in this emerging class of language models.
  arXiv  Detail & Related papers  (2025-07-15T08:44:46Z)
• Decompiling Smart Contracts with a Large Language Model [51.49197239479266]
  Despite Etherscan's 78,047,845 smart contracts deployed on (as of May 26, 2025), a mere 767,520 ( 1%) are open source.<n>This opacity necessitates the automated semantic analysis of on-chain smart contract bytecode.<n>We introduce a pioneering decompilation pipeline that transforms bytecode into human-readable and semantically faithful Solidity code.
  arXiv  Detail & Related papers  (2025-06-24T13:42:59Z)
• ReF Decompile: Relabeling and Function Call Enhanced Decompile [50.86228893636785]
  The goal of decompilation is to convert compiled low-level code (e.g., assembly code) back into high-level programming languages.<n>This task supports various reverse engineering applications, such as vulnerability identification, malware analysis, and legacy software migration.
  arXiv  Detail & Related papers  (2025-02-17T12:38:57Z)
• Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing [107.97160023681184]
  Aligned large language models (LLMs) are vulnerable to jailbreaking attacks. We propose SEMANTICSMOOTH, a smoothing-based defense that aggregates predictions of semantically transformed copies of a given input prompt.
  arXiv  Detail & Related papers  (2024-02-25T20:36:03Z)
• DRSM: De-Randomized Smoothing on Malware Classifier Providing Certified Robustness [58.23214712926585]
  We develop a certified defense, DRSM (De-Randomized Smoothed MalConv), by redesigning the de-randomized smoothing technique for the domain of malware detection. Specifically, we propose a window ablation scheme to provably limit the impact of adversarial bytes while maximally preserving local structures of the executables. We are the first to offer certified robustness in the realm of static detection of malware executables.
  arXiv  Detail & Related papers  (2023-03-20T17:25:22Z)
• Adversarial EXEmples: A Survey and Experimental Evaluation of Practical Attacks on Machine Learning for Windows Malware Detection [67.53296659361598]
  adversarial EXEmples can bypass machine learning-based detection by perturbing relatively few input bytes. We develop a unifying framework that does not only encompass and generalize previous attacks against machine-learning models, but also includes three novel attacks. These attacks, named Full DOS, Extend and Shift, inject the adversarial payload by respectively manipulating the DOS header, extending it, and shifting the content of the first section.
  arXiv  Detail & Related papers  (2020-08-17T07:16:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.