Meganeura: Portable GPU Training and Inference through Vulkan and Metal

• URL: http://arxiv.org/abs/2608.01563v1
• Date: Mon, 03 Aug 2026 00:42:55 GMT
• Title: Meganeura: Portable GPU Training and Inference through Vulkan and Metal
• Abstract summary: Meganeura asks whether one compact native compiler can span both phases on consumer GPU.<n>We compare five matched workloads with PyTorch on NVIDIA and AMD discrete GPUs.<n>The results show that general consumer graphics APIs can support a compact shared train-to-deploy stack at useful, sometimes vendor-competitive performance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training and deployed inference often cross export, conversion, and platform-specific runtime boundaries. Meganeura asks whether one compact native compiler can span both phases on consumer GPUs. Its typed static graph, automatic differentiation, optimizer, checkpoint, memory planner, and runtime lower specialized programs through Vulkan and Metal. We compare five matched workloads with PyTorch on NVIDIA and AMD discrete GPUs, an AMD APU, Apple silicon, and an Intel iGPU. The protocol separates strict f32 from validated fast paths and gates forward and backward independently. Forty-eight of 50 device-workload-mode cells pass both gates; the other two share one unresolved backward-reference disagreement on a newly supported APU. In strict f32, Meganeura wins 12 of 20 GPU-referenced minimal-latency cells and has a median valid training gap of 1.8x. On the discrete AMD GPU, four of five inference workloads are within 1.10x of compiled ROCm PyTorch and three training workloads are faster. Under accelerated contracts, the worst training gap is 4.6x. Compilation takes 0.1-2.4 seconds versus 6-96 seconds for torch.compile on supported GPU paths; the stripped binary is 13 MiB. Dispatch profiles localize the largest gaps to convolution derivatives and attention backward. A physical Android XR case study transfers a Meganeura-trained decoder into an Adreno/OpenXR application sharing the graphics queue. The results show that general consumer graphics APIs can support a compact shared train-to-deploy stack at useful, sometimes vendor-competitive performance. The measured gaps point to kernel coverage, scheduling, and arithmetic policy rather than an identified API limitation.

Related papers

• Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models [67.17268530365189]
  This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models.<n>It combines and specializes various existing and novel parallelism techniques at different layers and stages of the MoE model training pipeline.
  arXiv  Detail & Related papers  (2026-07-02T08:06:57Z)
• Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers [0.0]
  WebGPU's security-focused design imposes per-operation validation that compounds across the many small dispatches in neural network inference.<n>We present a systematic characterization of WebGPU dispatch overhead for LLM inference at batch size 1, spanning four vendors ( NVIDIA, AMD, Apple, Intel), two native implementations (Dawn, wgpu-native) and three browsers (Chrome, Safari, Firefox)<n>Our primary contribution is a sequential-dispatch methodology that reveals naive single-operation benchmarks overestimate dispatch by bysimtimes$.
  arXiv  Detail & Related papers  (2026-02-09T20:14:42Z)
• Characterization of GPU TEE Overheads in Distributed Data Parallel ML Training [7.236249885667945]
  Confidential computing (CC) or trusted execution enclaves (TEEs) is now the most common approach to enable secure computing in the cloud.<n>Recent introduction of GPU TEEs by NVIDIA enables machine learning (ML) models to be trained without leaking model weights or data to the cloud provider.<n>We present an in-depth characterization study on performance overhead associated with running distributed data parallel (DDP) ML training with GPU TEEs.
  arXiv  Detail & Related papers  (2025-01-20T22:23:50Z)
• Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading [2.8231000588510757]
  Transformers and large language models(LLMs) have seen rapid adoption in all domains. Training of transformers is very expensive and often hits a memory wall'' We propose a novel technique to split the LLM into subgroups, whose update phase is scheduled on either the CPU or the GPU.
  arXiv  Detail & Related papers  (2024-10-26T00:43:59Z)
• Scaling Tractable Probabilistic Circuits: A Systems Perspective [53.76194929291088]
  PyJuice is a general implementation design for PCs that improves prior art in several regards.<n>It is 1-2 orders of magnitude faster than existing systems at training large-scale PCs.<n>PyJuice consumes 2-5x less memory, which enables us to train larger models.
  arXiv  Detail & Related papers  (2024-06-02T14:57:00Z)
• PockEngine: Sparse and Efficient Fine-tuning in a Pocket [62.955793932377524]
  We introduce PockEngine: a tiny, sparse and efficient engine to enable fine-tuning on various edge devices. PockEngine supports sparse backpropagation and sparsely updates the model with measured memory saving and latency reduction. Remarkably, PockEngine enables fine-tuning LLaMav2-7B on NVIDIA Jetson AGX Orin at 550 tokens/s, 7.9$times$ faster than the PyTorch.
  arXiv  Detail & Related papers  (2023-10-26T19:46:11Z)
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU [89.2451963569343]
  FlexGen is a generation engine for running large language model (LLM) inference on a single commodity GPU. When running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours.
  arXiv  Detail & Related papers  (2023-03-13T05:19:28Z)
• Heterogeneous Acceleration Pipeline for Recommendation System Training [1.8457649813040096]
  Recommendation models rely on deep learning networks and large embedding tables. These models are typically trained using hybrid-GPU or GPU-only configurations. This paper introduces Hotline, a heterogeneous CPU acceleration pipeline.
  arXiv  Detail & Related papers  (2022-04-11T23:10:41Z)
• PLSSVM: A (multi-)GPGPU-accelerated Least Squares Support Vector Machine [68.8204255655161]
  Support Vector Machines (SVMs) are widely used in machine learning. However, even modern and optimized implementations do not scale well for large non-trivial dense data sets on cutting-edge hardware. PLSSVM can be used as a drop-in replacement for an LVM.
  arXiv  Detail & Related papers  (2022-02-25T13:24:23Z)
• UNIT: Unifying Tensorized Instruction Compilation [11.193044425743981]
  Hardware vendors offer tensorized instructions for mixed-precision operations, like Intel VNNI, Core, and ARM-DOT. The lack of compilation techniques for this makes it hard to utilize these instructions. We develop a compiler framework to unify the compilation for these instructions.
  arXiv  Detail & Related papers  (2021-01-21T06:22:58Z)
• Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learning [7.43260596107574]
  We propose Nimble, a deep learning (DL) execution engine that runs tasks in parallel with minimal scheduling overhead. Nable automatically parallelizes the execution of GPU tasks by exploiting multiple GPU streams in a single GPU. evaluation on a variety of neural networks shows that compared to PyTorch, Nimble speeds up inference and training by up to 22.34$times$ and 3.61$times$, respectively.
  arXiv  Detail & Related papers  (2020-12-04T17:25:46Z)
• Hybrid Models for Learning to Branch [81.93868699246214]
  We propose a new hybrid architecture for efficient branching on CPU machines. The proposed architecture combines the expressive power of GNNs with computationally inexpensive multi-layer perceptrons (MLP) for branching.
  arXiv  Detail & Related papers  (2020-06-26T21:03:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.