Native LLM and MLLM Inference at Scale on Apple Silicon

• URL: http://arxiv.org/abs/2601.19139v2
• Date: Thu, 29 Jan 2026 06:19:19 GMT
• Title: Native LLM and MLLM Inference at Scale on Apple Silicon
• Abstract summary: We present vllm-mlx, a framework for efficient LLM and MLLM inference on Apple Silicon built native on MLX.<n>For text models, we achieve 21% to 87% higher throughput than llama- across models ranging from Qwen3-0.6B to Nemotron-30B.<n>For multimodal models, we introduce content-based caching that eliminates redundant vision encoding by identifying identical images through content hashing, regardless of input format.
• Score: 0.8122270502556375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The growing adoption of Apple Silicon for machine learning development has created demand for efficient inference solutions that leverage its unique unified memory architecture. However, existing tools either lack native optimization (PyTorch MPS) or focus solely on text models, leaving multimodal workloads underserved. We present vllm-mlx, a framework for efficient LLM and MLLM inference on Apple Silicon built natively on MLX. For text models, we achieve 21\% to 87\% higher throughput than llama-cpp across models ranging from Qwen3-0.6B to Nemotron-30B, while providing continuous batching that scales to 4.3x aggregate throughput at 16 concurrent requests. For multimodal models, we introduce content-based prefix caching that eliminates redundant vision encoding by identifying identical images through content hashing, regardless of input format. Our evaluation on Apple M4 Max demonstrates throughput of up to 525 tokens per second on text models and 28x speedup on repeated image queries, reducing multimodal latency from 21.7 seconds to under 1 second. Video analysis with up to 64 frames achieves 24.7x cache speedup. We release our implementation as open source to support efficient inference on consumer Apple hardware.

Related papers

• NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference [1.3961509337714324]
  We introduce NeoMME, a family of 260M and 800M Multimodal and bidirectional Multilingual document encoders.<n>Both models are pretrained from scratch with a masked discrete-diffusion text objective.<n>On ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models with 0.523 nDCG@10.<n>At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT.
  arXiv  Detail & Related papers  (2026-08-31T13:08:15Z)
• BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal [0.0]
  We present BaseRT, a native Metal runtime for large language models (LLMs) on Apple.<n>Report the highest inference throughput on this hardware to date.
  arXiv  Detail & Related papers  (2026-07-01T06:37:11Z)
• Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM [6.75883098679462]
  We present a hardware-aware framework for efficient on-device inference of a LLaMA-based multilingual foundation model on Samsung Galaxy S24 and S25 devices.<n>Our system achieves 4-6x overall improvements in memory and latency while maintaining accuracy across 9 languages and 8 tasks.
  arXiv  Detail & Related papers  (2026-04-20T07:36:20Z)
• Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge [9.701124246177661]
  We investigate the feasibility of using Multimodal Large Language Models (MLLMs) for real-time online episodic memory question answering.<n>Experiments on the QAEgo4D-Closed benchmark analyze the performance of Multimodal Large Language Models (MLLMs) within strict resource boundaries.
  arXiv  Detail & Related papers  (2026-02-25T22:30:56Z)
• AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model [40.488271586857884]
  AndesVL is a suite of mobile-side MLLMs with 0.6B to 4B parameters based on Qwen3's LLM and various visual encoders.<n>We introduce a 1+N LoRA architecture alongside a Quantization-Aware LoRA Fine-Tuning framework to facilitate efficient task adaptation and model compression.<n>We achieve a 6.7x peak decoding speedup ratio, up to 30.9% memory reduction, and 1.8 bits-per-weight when deploying AndesVL-4B on MediaTek Dimensity 9500 chips.
  arXiv  Detail & Related papers  (2025-10-13T15:04:38Z)
• Production-Grade Local LLM Inference on Apple Silicon: A Comparative Study of MLX, MLC-LLM, Ollama, llama.cpp, and PyTorch MPS [0.08030359871216612]
  We present a systematic, empirical evaluation of five local large language models (LLM)s on Apple Silicon.<n>MLX, MLC-LLM, llama, Ollama, and PyTorch MPS were tested.
  arXiv  Detail & Related papers  (2025-10-09T23:53:38Z)
• dParallel: Learnable Parallel Decoding for dLLMs [77.24184219948337]
  Diffusion large language models (dLLMs) offer parallel token prediction and lower inference latency.<n>Existing open-source models still require nearly token-length decoding steps to ensure performance.<n>We introduce dParallel, a simple and effective method that unlocks the inherent parallelism of dLLMs for fast sampling.
  arXiv  Detail & Related papers  (2025-09-30T16:32:52Z)
• Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation [108.0657508755532]
  We introduce the Gated Memory Unit (GMU), a simple yet effective mechanism for efficient memory sharing across layers.<n>We apply it to create SambaY, a decoder-hybrid-decoder architecture that incorporates GMUs to share memory readout states from a Samba-based self-decoder.
  arXiv  Detail & Related papers  (2025-07-09T07:27:00Z)
• AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding [55.320254859515714]
  Multimodal Large Language Models (MLLMs) have revolutionized video understanding, yet are still limited by context length when processing long videos.<n>We propose AdaReTaKe, a training-free method that flexibly reduces visual redundancy by allocating compression ratios among time and layers with theoretical guarantees.<n>Experiments on VideoMME, MLVU, LongVideoBench, and LVBench datasets demonstrate that AdaReTaKe outperforms existing methods by 2.3% and 2.8% for 7B and 72B models, respectively.
  arXiv  Detail & Related papers  (2025-03-16T16:14:52Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs [15.276687781165608]
  Large Language Model (LLM) is widely employed for tasks such as intelligent assistants, text summarization, translation, and multi-modality on mobile phones. To facilitate high-efficiency LLM deployment on device GPU, we propose four optimization techniques. We implement these techniques in our mobile inference engine, Transformer-Lite, which is compatible with both Qualcomm and MTK processors.
  arXiv  Detail & Related papers  (2024-03-29T08:26:53Z)
• MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices [73.46317110474064]
  MobileVLM is a competent multimodal vision language model (MMVLM) targeted to run on mobile devices. It comprises a set of language models at the scale of 1.4B and 2.7B parameters, trained from scratch, a multimodal vision model that is pre-trained in the CLIP fashion.
  arXiv  Detail & Related papers  (2023-12-28T08:21:24Z)
• DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training [82.06732962485754]
  FlashAttention effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. We introduce DISTFLASHATTN, a memory-efficient attention mechanism optimized for long-context LLMs training. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses.
  arXiv  Detail & Related papers  (2023-10-05T03:47:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.