BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

• URL: http://arxiv.org/abs/2607.19438v1
• Date: Tue, 21 Jul 2026 06:42:18 GMT
• Title: BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators
• Abstract summary: We show that BaseRT exploits the Metal inference for large language models on Apple Silicon.<n>On an Apple M5 Pro, BaseRT delivers up to $6.4times$ higher prompt-processing throughput than llama.<n>We show that the M5's tensor cores are the decisive lever for prompt processing on Apple Silicon.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Apple's M5 generation introduces a redesigned GPU architecture in which every core carries a dedicated Neural Accelerator: on-die matrix units exposed through the Metal~4 tensor API. We show that BaseRT, our native Metal inference runtime for large language models on Apple Silicon, exploits these units to push inference throughput on Apple hardware substantially beyond both llama.cpp and MLX. Building on BaseRT's framework-free design, we add a family of hand-written Metal~4 tensor-core kernels (including dense and mixture-of-experts GEMM and flash-attention prefill kernels) that route the compute-bound matrix multiplications of inference through the M5 Neural Accelerators while leaving the memory-bound decode path on our existing specialised kernels. On an Apple M5 Pro, across fifteen model configurations spanning the Qwen3, Qwen3.5/3.6, Llama~3.2, and Gemma~4 families from sub-1B to 35B parameters, BaseRT delivers up to $6.4\times$ higher prompt-processing throughput than llama.cpp and $3.9\times$ higher than MLX, with the largest margins on the mixture-of-experts models where matrix multiplication dominates, while maintaining its lead on decode of up to $1.75\times$ over llama.cpp and $1.33\times$ over MLX. These results establish a new performance ceiling for on-device LLM inference and show that the M5's tensor cores are the decisive lever for prompt processing on Apple Silicon. BaseRT is publicly available at https://github.com/basecompute/baseRT.

Related papers

• BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal [0.0]
  We present BaseRT, a native Metal runtime for large language models (LLMs) on Apple.<n>Report the highest inference throughput on this hardware to date.
  arXiv  Detail & Related papers  (2026-07-01T06:37:11Z)
• Rigel: Reverse-Engineering the Metal 4.1 Tensor Compute Path on the Apple M4 Max GPU [3.151184728006369]
  We present Rigel, an empirical characterization of this path on a single Apple M4 Max.<n>We reconstruct the opaque 8x8 cooperative_tensor fragment layout Apple documents nowhere.<n>Acting on the characterization, a hand-fused GEMM + bias + GELU kernel beats the decomposed path by +6.5-12.9% in the cache-resident regime.
  arXiv  Detail & Related papers  (2026-06-11T00:10:23Z)
• AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference [58.285369364699]
  We present AgentCompile, an LLM-guided inference compiler that uses LLM outputs only as advisory search metadata.<n>Given compiler-derived region summaries and bounded candidate spaces, the LLM proposes semantic labels, candidate priorities, parameter hints, and risk annotations.<n>AgentCompile materializes candidates through graphs templates, checks interface and hardware constraints, validates candidates empirically, selects implementations by measured latency, and falls back when specialization is unsupported or unprofitable.<n>In end-to-end autoregressive generation, AgentCompile averages 5.66x, 4.05x, and 4.26x speedup over PyTorch on Qwen3-1.7B, Qwen
  arXiv  Detail & Related papers  (2026-06-04T03:49:56Z)
• Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon [6.599344783327054]
  Metal-Sci is a 10-task benchmark of scientific Apple Silicon Metal compute kernels.<n>Each task ships a CPU reference, a roofline-anchored fitness function, and a held-out generalization size.<n>We report matched single-model sweeps of Claude Opus 4.7, Gemini 3.1 Pro, and GPT 5.5 on M1 Pro.
  arXiv  Detail & Related papers  (2026-05-10T19:11:26Z)
• mlx-snn: Spiking Neural Networks on Apple Silicon via MLX [1.1597621848542221]
  We introduce mlx-snn, the first spiking neural network (SNN) library built on Apple's MLX framework.<n>mlx-snn provides six neuron models, four surrogate gradient functions, four spike encoding methods, and a complete backpropagation-through-time training pipeline.<n>mlx-snn is open-source under the MIT license and available on PyPI.
  arXiv  Detail & Related papers  (2026-03-03T21:25:36Z)
• Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS [0.08030359871216612]
  We present a systematic, empirical evaluation of five local large language models (LLM)s on Apple Silicon.<n>MLX, MLC-LLM, llama, Ollama, and PyTorch MPS were tested.
  arXiv  Detail & Related papers  (2025-10-09T23:53:38Z)
• Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers [65.35142508909892]
  We present a novel four-dimensional hybrid parallel algorithm implemented in a highly scalable, portable, open-source framework called AxoNN.<n>We demonstrate fine-tuning of a 405-billion parameter LLM using AxoNN on Frontier.
  arXiv  Detail & Related papers  (2025-02-12T06:05:52Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• Recurrent Drafter for Fast Speculative Decoding in Large Language Models [18.342742904042673]
  We present Recurrent Drafter, an advanced speculative decoding approach.<n>It achieves state-of-the-art speedup for large language models (LLMs) inference.
  arXiv  Detail & Related papers  (2024-03-14T23:40:56Z)
• INR-Arch: A Dataflow Architecture and Compiler for Arbitrary-Order Gradient Computations in Implicit Neural Representation Processing [66.00729477511219]
  Given a function represented as a computation graph, traditional architectures face challenges in efficiently computing its nth-order gradient. We introduce INR-Arch, a framework that transforms the computation graph of an nth-order gradient into a hardware-optimized dataflow architecture. We present results that demonstrate 1.8-4.8x and 1.5-3.6x speedup compared to CPU and GPU baselines respectively.
  arXiv  Detail & Related papers  (2023-08-11T04:24:39Z)
• Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts [60.1586169973792]
  M$3$ViT is the latest multi-task ViT model that introduces mixture-of-experts (MoE) MoE achieves better accuracy and over 80% reduction computation but leaves challenges for efficient deployment on FPGA. Our work, dubbed Edge-MoE, solves the challenges to introduce the first end-to-end FPGA accelerator for multi-task ViT with a collection of architectural innovations.
  arXiv  Detail & Related papers  (2023-05-30T02:24:03Z)
• Hybrid Models for Learning to Branch [81.93868699246214]
  We propose a new hybrid architecture for efficient branching on CPU machines. The proposed architecture combines the expressive power of GNNs with computationally inexpensive multi-layer perceptrons (MLP) for branching.
  arXiv  Detail & Related papers  (2020-06-26T21:03:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.