MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale

• URL: http://arxiv.org/abs/2603.15954v1
• Date: Mon, 16 Mar 2026 22:10:50 GMT
• Title: MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale
• Abstract summary: Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware.<n>We present a methodology for designing such models using hardware-in-the-loop architecture search under mobile latency constraints.
• Score: 36.89558970450915
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce near-real-time responses and exhibit broad hardware compatibility, maximizing user reach. We present a methodology for designing such models using hardware-in-the-loop architecture search under mobile latency constraints. This system is amenable to industry-scale deployment: it generates models deployable without custom kernels and compatible with standard mobile runtimes like Executorch. Our methodology avoids specialized attention mechanisms and instead uses attention skipping for long-context acceleration. Our approach jointly optimizes model architecture (layers, dimensions) and attention pattern. To efficiently evaluate candidates, we treat each as a pruned version of a pretrained backbone with inherited weights, thereby achieving high accuracy with minimal continued pretraining. We leverage the low cost of latency evaluation in a staged process: learning an accurate latency model first, then searching for the Pareto-frontier across latency and quality. This yields MobileLLM-Flash, a family of foundation models (350M, 650M, 1.4B) for efficient on-device use with strong capabilities, supporting up to 8k context length. MobileLLM-Flash delivers up to 1.8x and 1.6x faster prefill and decode on mobile CPUs with comparable or superior quality. Our analysis of Pareto-frontier design choices offers actionable principles for OD-LLM design.

Related papers

• Characterizing VLA Models: Identifying the Action Generation Bottleneck for Edge AI Architectures [4.6111834169518735]
  Vision-Language-Action (VLA) models are an emerging class of workloads critical for robotics and embodied AI at the edge.<n>This paper characterizes VLA performance on two generations of edge hardware, viz. the Nvidia Jetson Orin and Thor platforms.
  arXiv  Detail & Related papers  (2026-03-01T01:09:55Z)
• Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs [49.99513618431772]
  We propose a hardware co-design law that captures model accuracy and inference performance.<n>We empirically evaluate 1,942 candidate architectures on NVIDIA Jetson Orin.<n>Our architecture achieves 19.42% lower perplexity on WikiText-2.
  arXiv  Detail & Related papers  (2026-02-10T23:51:00Z)
• Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models [97.55009021098554]
  This work aims to identify the key determinants of SLMs' real-device latency and offer generalizable principles and methodologies for SLM design and training.<n>We introduce a new family of hybrid SLMs, called Nemotron-Flash, which significantly advances the accuracy-efficiency frontier of state-of-the-art SLMs.
  arXiv  Detail & Related papers  (2025-11-24T08:46:36Z)
• MOBIUS: Big-to-Mobile Universal Instance Segmentation via Multi-modal Bottleneck Fusion and Calibrated Decoder Pruning [91.90342432541138]
  Scaling up model size and training data has advanced foundation models for instance-level perception.<n>High computational cost limits adoption on resource-constrained platforms.<n>We introduce a new benchmark for efficient segmentation on both high-performance computing platforms and mobile devices.
  arXiv  Detail & Related papers  (2025-10-16T18:00:00Z)
• LongCat-Flash Technical Report [165.64670448930875]
  LongCat-Flash is a 560-billion- parameter Mixture-of-Experts (MoE) language model.<n>It is designed for both computational efficiency and advanced agentic capabilities.<n>We complete model training on more than 20 trillion tokens within 30 days, while achieving over 100 tokens per second (TPS) for inference at a cost of $0.70 per million output tokens.
  arXiv  Detail & Related papers  (2025-09-01T10:05:45Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)
• MELTing point: Mobile Evaluation of Language Transformers [8.238355633015068]
  We explore the current state of mobile execution of Large Language Models (LLMs) We have created our own automation infrastructure, MELT, which supports the headless execution and benchmarking of LLMs on device. We evaluate popular instruction fine-tuned LLMs and leverage different frameworks to measure their end-to-end and granular performance.
  arXiv  Detail & Related papers  (2024-03-19T15:51:21Z)
• Moses: Efficient Exploitation of Cross-device Transferable Features for Tensor Program Optimization [10.115260534967645]
  We propose Moses, a simple and efficient design based on the lottery ticket hypothesis. Compared with state-of-the-art approaches, Moses achieves up to 1.53X efficiency gain in the search stage.
  arXiv  Detail & Related papers  (2022-01-15T03:55:52Z)
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale [11.121380180647769]
  We share in this paper our search strategies to adapt reference recommendation models to low-precision hardware. We also discuss the design and development of tool chain so as to maintain our models' accuracy throughout their lifespan. We believe these lessons from the trenches promote better co-design between hardware architecture and software engineering.
  arXiv  Detail & Related papers  (2021-05-26T16:42:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.