DVM: A Bytecode Virtual Machine Approach for Dynamic Tensor Computation

• URL: http://arxiv.org/abs/2603.24239v2
• Date: Thu, 02 Apr 2026 02:49:12 GMT
• Title: DVM: A Bytecode Virtual Machine Approach for Dynamic Tensor Computation
• Abstract summary: Dynamism is common in AI computation, e.g., the dynamic tensor shapes and the dynamic control flows in models.<n>Due to the long compilation time, existing runtime compilation damages the model efficiency.<n>We propose a real-time compiler, DVM, to speed up the compilation or hide the compilation overhead.
• Score: 6.872249333818181
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Dynamism is common in AI computation, e.g., the dynamic tensor shapes and the dynamic control flows in models. Due to the long compilation time, existing runtime compilation damages the model efficiency, while the offline compilers either suffer from the long compilation time and device memory footprint to cover all the possible execution instances of a dynamic model, or sacrifice optimization opportunities for usability. In this paper, we rethink the feasibility of runtime compilation for dynamic models and identify that the key for it to work is to speed up the compilation or hide the compilation overhead. To do this, we propose a real-time compiler, DVM. In DVM, we design a runtime operator compiler based on a bytecode virtual machine to perform effective and efficient compilation for each dynamic operator instance given its input. Specifically, instead of compiling programs into machine code, we encode the operator program into bytecode on the CPU and decode the bytecode into virtual instructions for direct execution on the NPU. Based on the runtime operator compiler, we further propose an operator fuser, which performs symbol-deduction-based fusion on static graphs and runtime fusion on dynamic graphs. Both pattern- and stacking-based fusion are supported to increase fusion opportunities. Evaluation on operators, subgraphs, and models shows that, compared with TorchInductor, PyTorch-eager and MindSpore-graph-O0, we are up to 11.77$\times$ better in terms of the operator/model efficiency and up to 5 orders of magnitude faster in terms of the maximum compilation time.

Related papers

• AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Code [69.84692701646769]
  AutoVecCoder is a novel framework designed to empower Large Language Models with the capability of automated explicit vectorization.<n>VecPrompt, an automated data synthesis pipeline, injects domain-specific intrinsic knowledge.<n>VecRL, a reinforcement learning framework, aligns code generation with execution efficiency.<n>AutoVecCoder-8B trained by this framework achieves state-of-the-art performance on the SSE and AVX subsets of SimdBench.
  arXiv  Detail & Related papers  (2026-05-18T07:33:15Z)
• WGRAMMAR: Leverage Prior Knowledge to Accelerate Structured Decoding [58.1177179119881]
  We introduce wgrammar, a lightweight decoding engine that integrates domain-aware simplification, constraint decomposition, and mask caching.<n> wgrammar achieves up to 250x speedup over existing systems.
  arXiv  Detail & Related papers  (2025-07-22T17:13:47Z)
• TileLang: A Composable Tiled Programming Model for AI Systems [17.240134151647187]
  We present TileLang, a generalized tiled programming model for more efficient AI programming.<n> TileLang decouples scheduling space (thread binding, layout, tensorize and pipeline) from dataflow, and encapsulated them as a set of customization annotations and primitives.<n>We conduct comprehensive experiments on commonly-used devices, across numerous experiments, our evaluation shows that TileLang can achieve state-of-the-art performance in key kernels.
  arXiv  Detail & Related papers  (2025-04-24T14:08:49Z)
• Hexcute: A Tile-based Programming Language with Automatic Layout and Task-Mapping Synthesis [8.742879659920643]
  Hexcute is a tile-based programming language that exposes shared memory and register abstractions to enable fine-grained optimization for mixed-type operators.<n>It automates layout and task mapping synthesis with a novel type-inference-based algorithm.<n>Our evaluation shows that Hexcute generalizes to a wide range of DL operators, achieves 1.7-11.28$times$ speedup over existing DL compilers for mixed-type operators, and brings up to 2.91$times$ speedup in the end-to-end evaluation.
  arXiv  Detail & Related papers  (2025-04-22T19:01:28Z)
• Tempo: Compiled Dynamic Deep Learning with Symbolic Dependence Graphs [0.2578242050187029]
  We describe Tempo, a new deep learning system that combines the dynamism of eager execution with the whole-program optimizations of graph-based compilation.<n>We show that Tempo achieves a 7$times$ speedup over JAX for Llama-3.2-3B decoding.<n>For reinforcement learning algorithms, Tempo achieves a 54$times$ speedup, with 16$times$ lower peak memory usage.
  arXiv  Detail & Related papers  (2025-01-09T18:05:33Z)
• PowerFusion: A Tensor Compiler with Explicit Data Movement Description and Instruction-level Graph IR [10.059491353103526]
  We propose IntelliGen, a tensor compiler that can generate high-performance code for memory-intensive operators. IntelliGen considers both computation and data movement optimizations. We evaluate IntelliGen on NVIDIA GPU, AMD GPU, and Cambricon MLU, showing speedup up to 1.97x, 2.93x, and 16.91x (1.28x, 1.23x, and 2.31x on average)
  arXiv  Detail & Related papers  (2023-07-11T03:17:40Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)
• Learning Performance-Improving Code Edits [107.21538852090208]
  We introduce a framework for adapting large language models (LLMs) to high-level program optimization. First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs. For prompting, we propose retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play.
  arXiv  Detail & Related papers  (2023-02-15T18:59:21Z)
• LoopStack: a Lightweight Tensor Algebra Compiler Stack [61.04098601022665]
  LoopStack is a domain specific compiler stack for tensor operations. It generates machine code that matches and frequently exceeds the performance of in state-of-the-art machine learning frameworks. It has a very small memory footprint - a binary size of 245KB, and under 30K lines of effective code makes it ideal for use on mobile and embedded devices.
  arXiv  Detail & Related papers  (2022-05-02T01:57:58Z)
• High-performance symbolic-numerics via multiple dispatch [52.77024349608834]
  Symbolics.jl is an extendable symbolic system which uses dynamic multiple dispatch to change behavior depending on the domain needs. We show that by formalizing a generic API on actions independent of implementation, we can retroactively add optimized data structures to our system. We demonstrate the ability to swap between classical term-rewriting simplifiers and e-graph-based term-rewriting simplifiers.
  arXiv  Detail & Related papers  (2021-05-09T14:22:43Z)
• UNIT: Unifying Tensorized Instruction Compilation [11.193044425743981]
  Hardware vendors offer tensorized instructions for mixed-precision operations, like Intel VNNI, Core, and ARM-DOT. The lack of compilation techniques for this makes it hard to utilize these instructions. We develop a compiler framework to unify the compilation for these instructions.
  arXiv  Detail & Related papers  (2021-01-21T06:22:58Z)
• PolyDL: Polyhedral Optimizations for Creation of High Performance DL primitives [55.79741270235602]
  We present compiler algorithms to automatically generate high performance implementations of Deep Learning primitives. We develop novel data reuse analysis algorithms using the polyhedral model. We also show that such a hybrid compiler plus a minimal library-use approach results in state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-06-02T06:44:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.