BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

• URL: http://arxiv.org/abs/2607.00501v1
• Date: Wed, 01 Jul 2026 06:37:11 GMT
• Title: BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal
• Abstract summary: We present BaseRT, a native Metal runtime for large language models (LLMs) on Apple.<n>Report the highest inference throughput on this hardware to date.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present BaseRT, a native Metal inference runtime for large language models (LLMs) on Apple Silicon, and report the highest inference throughput on this hardware to date. Existing runtimes, including llama.cpp and MLX-based frameworks, incur overhead from abstractions not designed for Metal's execution model or Apple Silicon's unified memory topology. By building natively on Metal with chip-specific kernel fusion, unified memory-aware optimisation, and custom dispatch logic, BaseRT recovers performance that framework-based approaches leave on the table. BaseRT supports a wide range of model families across eight quantisation formats (Q2 to FP16) on all Apple M-series devices. In this paper, we evaluate the Qwen3, Llama 3.2, and Gemma 4 families at Q4 and Q8 quantisation on M3 and M4 Pro devices. BaseRT achieves up to 1.56x higher decode throughput than llama.cpp and up to 1.35x higher than MLX, with substantially larger margins on prefill for mixture-of-experts models, delivering consistent best-in-class throughput from sub-1B to 30B parameter models. These results establish Apple Silicon as a more capable inference platform than previously reported, with direct implications for the emerging edge inference paradigm: as privacy requirements, latency constraints, and cloud cost pressures drive inference toward on-device deployment, performance-optimised local runtimes are a critical enabling layer for this transition. BaseRT is publicly available at https://github.com/basecompute/baseRT

Related papers

• Opt.Gear Technical Report [3.573729952898132]
  OptGear is a foundation model designed for efficient on-device deployment, real-tim inference, and strong task capability.<n>It includes a dense model (1M, 270M, and 1B) with a context length of 64K.<n>OptGear-1M is the first generative language model to achieve 20 TPS with W4A32 quantization on the ARM Cortex-M7 CPU.
  arXiv  Detail & Related papers  (2026-08-02T06:43:25Z)
• BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators [0.0]
  We show that BaseRT exploits the Metal inference for large language models on Apple Silicon.<n>On an Apple M5 Pro, BaseRT delivers up to $6.4times$ higher prompt-processing throughput than llama.<n>We show that the M5's tensor cores are the decisive lever for prompt processing on Apple Silicon.
  arXiv  Detail & Related papers  (2026-07-21T06:42:18Z)
• StereoFactory: A Unified Merging Framework for Robust Stereo Matching [61.973843344605655]
  Stereo matching has advanced through foundation models trained on large-scale datasets, yet this paradigm suffers from a scalability bottleneck.<n>Model merging offers a scalable post-hoc alternative by integrating knowledge from specialized models after source checkpoints are available.<n>We propose StereoFactory, a coarse-to-fine evolutionary framework for adaptive model merging.
  arXiv  Detail & Related papers  (2026-06-16T03:36:59Z)
• Compiler-First State Space Duality and Portable $O(1)$ Autoregressive Caching for Inference [0.0]
  Mamba-2's state space duality algorithm maps cleanly onto what XLA's fusion and tiling passes actually optimise.<n>We implement the full inference path (prefill, cached autoregressive decoding) as shaped standard primitives under XLA.<n>The implementation runs unmodified on CPU, NVIDIA GPU, and Google Cloud TPU from a single JAX source.
  arXiv  Detail & Related papers  (2026-03-10T12:03:00Z)
• Native LLM and MLLM Inference at Scale on Apple Silicon [0.8122270502556375]
  We present vllm-mlx, a framework for efficient LLM and MLLM inference on Apple Silicon built native on MLX.<n>For text models, we achieve 21% to 87% higher throughput than llama- across models ranging from Qwen3-0.6B to Nemotron-30B.<n>For multimodal models, we introduce content-based caching that eliminates redundant vision encoding by identifying identical images through content hashing, regardless of input format.
  arXiv  Detail & Related papers  (2026-01-27T03:11:02Z)
• Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS [0.08030359871216612]
  We present a systematic, empirical evaluation of five local large language models (LLM)s on Apple Silicon.<n>MLX, MLC-LLM, llama, Ollama, and PyTorch MPS were tested.
  arXiv  Detail & Related papers  (2025-10-09T23:53:38Z)
• dInfer: An Efficient Inference Framework for Diffusion Language Models [54.80918957287927]
  Diffusion-based large language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs.<n>We present dInfer, an efficient and framework for dLLM inference.
  arXiv  Detail & Related papers  (2025-10-09T16:19:42Z)
• Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters [43.52299612686755]
  On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low throughput and capability.<n>We present prima., a distributed on-device inference system that runs 30-70B LLMs on consumer home clusters with mixed CPUs/GPUs, insufficient RAM/VRAM, slow disks, Wi-Fi links, and heterogeneous OSs.<n>We introduce pipelined-ring parallelism (PRP) to overlap disk I/O with compute and communication, and address the prefetch-release conflict in mmap-based offloading.
  arXiv  Detail & Related papers  (2025-04-07T13:46:21Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees [19.58773369944074]
  Finetuning large language models (LLMs) is essential for task adaptation, yet today's serving stacks isolate inference and finetuning on separate GPU clusters.<n>We introduce FlexLLM, the first system to co-serve LLM inference and PEFT-based finetuning on shared GPUs by fusing at the token level.<n>At runtime, a novel token-level finetuning mechanism paired with a hybrid token scheduler dynamically interleaves inference and training tokens within each co-serving iteration.
  arXiv  Detail & Related papers  (2024-02-29T01:33:08Z)
• Green AI: A Preliminary Empirical Study on Energy Consumption in DL Models Across Different Runtime Infrastructures [56.200335252600354]
  It is common practice to deploy pre-trained models on environments distinct from their native development settings. This led to the introduction of interchange formats such as ONNX, which includes its infrastructure, and ONNX, which work as standard formats.
  arXiv  Detail & Related papers  (2024-02-21T09:18:44Z)
• QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models [64.34635279436054]
  Mixture-of-Experts (MoE) architectures offer a general solution to the high inference costs of large language models (LLMs) via sparse routing. We present a solution to this memory problem, in form of a new compression and execution framework called QMoE.
  arXiv  Detail & Related papers  (2023-10-25T17:24:53Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.