MobileNMT: Enabling Translation in 15MB and 30ms

• URL: http://arxiv.org/abs/2306.04235v1
• Date: Wed, 7 Jun 2023 08:25:51 GMT
• Title: MobileNMT: Enabling Translation in 15MB and 30ms
• Abstract summary: We present MobileNMT, a system that can translate in 15MB and 30ms on devices. With the co-design of model and engine, compared with the existing system, we speed up 47.0x and save 99.5% of memory with only 11.6% loss of BLEU.
• Score: 53.75988363281843
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying NMT models on mobile devices is essential for privacy, low latency, and offline scenarios. For high model capacity, NMT models are rather large. Running these models on devices is challenging with limited storage, memory, computation, and power consumption. Existing work either only focuses on a single metric such as FLOPs or general engine which is not good at auto-regressive decoding. In this paper, we present MobileNMT, a system that can translate in 15MB and 30ms on devices. We propose a series of principles for model compression when combined with quantization. Further, we implement an engine that is friendly to INT8 and decoding. With the co-design of model and engine, compared with the existing system, we speed up 47.0x and save 99.5% of memory with only 11.6% loss of BLEU. The code is publicly available at https://github.com/zjersey/Lightseq-ARM.

Related papers

• Opt.Gear Technical Report [3.573729952898132]
  OptGear is a foundation model designed for efficient on-device deployment, real-tim inference, and strong task capability.<n>It includes a dense model (1M, 270M, and 1B) with a context length of 64K.<n>OptGear-1M is the first generative language model to achieve 20 TPS with W4A32 quantization on the ARM Cortex-M7 CPU.
  arXiv  Detail & Related papers  (2026-08-02T06:43:25Z)
• Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds [0.0]
  We present CROWDio, which achieves practical ONNX inference across resource-constrained Android workers without model modification.<n>Our system holds a peak per-device RSS to 43+-2 MB and limits battery draw to 50+-3 mAh per run, while streaming cuts 34% below barrier synchronisation.
  arXiv  Detail & Related papers  (2026-05-20T05:21:54Z)
• Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device [90.46496321553843]
  We present Mobile-O, a compact vision-language-diffusion model that brings unified multimodal intelligence to a mobile device.<n>Its core module, the Mobile Conditioning Projector (MCP), fuses vision-language features with a diffusion generator using depthwise-separable convolutions and layerwise alignment.<n>Running in only 3s per 512x512 image on an iPhone, Mobile-O establishes the first practical framework for real-time unified multimodal understanding and generation on edge devices.
  arXiv  Detail & Related papers  (2026-02-23T18:59:58Z)
• Tiny-QMoE [0.6875312133832079]
  The QMoE model provides a practical approach for compression of massive Mixture-of-Experts (MoE) models.<n>QMoE offers a solution geared towards memory limitations that often reach terabyte scales.
  arXiv  Detail & Related papers  (2025-09-26T21:33:20Z)
• Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices [12.40080721078945]
  Large Multimodal Models (LMMs) are inherently modular, consisting of vision and audio encoders, projectors, and large language models.<n>Yet, they are almost always executed monolithically, which underutilizes the heterogeneous accelerators.<n>We present a hardware-software co-design inference framework for Large Multimodal Models (LMMs)<n>This framework breaks large models into modular components and scheduled to run on the most appropriate compute units.
  arXiv  Detail & Related papers  (2025-09-25T22:28:44Z)
• SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment [5.141876811512978]
  SmallThinker is a family of large language models (LLMs) designed for local devices.<n>We introduce a two-level sparse structure combining fine-grained Mixture-of-Experts (MoE) with sparse feed-forward networks.<n>We release SmallThinker-4B-A0.6B and SmallThinker-21B-A3B, which achieve state-of-the-art performance scores and even outperform larger LLMs.
  arXiv  Detail & Related papers  (2025-07-28T16:45:14Z)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [71.43026659686679]
  Large Language Models (LLMs) have grown rapidly in size, creating challenges for efficient deployment on resource-constrained hardware. We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• Squat: Quant Small Language Models on the Edge [45.448118471829474]
  A growing trend has emerged in designing high-quality Small Language Models (SLMs) with a few million parameters.<n>Considering that full parameter training is feasible for SLMs on mobile devices, Quantization-Aware Training (QAT) is employed to improve efficiency.<n>We propose Squat method, an effective QAT framework with deployable quantization for SLMs on mobile devices.
  arXiv  Detail & Related papers  (2024-02-16T16:10:38Z)
• MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices [73.46317110474064]
  MobileVLM is a competent multimodal vision language model (MMVLM) targeted to run on mobile devices. It comprises a set of language models at the scale of 1.4B and 2.7B parameters, trained from scratch, a multimodal vision model that is pre-trained in the CLIP fashion.
  arXiv  Detail & Related papers  (2023-12-28T08:21:24Z)
• QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models [64.34635279436054]
  Mixture-of-Experts (MoE) architectures offer a general solution to the high inference costs of large language models (LLMs) via sparse routing. We present a solution to this memory problem, in form of a new compression and execution framework called QMoE.
  arXiv  Detail & Related papers  (2023-10-25T17:24:53Z)
• Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production [7.056223012587321]
  We introduce a highly efficient inference framework with several optimization approaches to accelerate the computation of sparse models. We are able to deploy 136x larger models with 27% less cost and significantly better quality compared to the existing solutions.
  arXiv  Detail & Related papers  (2022-11-18T03:43:52Z)
• MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning [114.66037224769005]
  We present a novel MicroISP model designed specifically for edge devices. The proposed solution is capable of processing up to 32MP photos on recent smartphones using the standard mobile ML libraries. The architecture of the model is flexible, allowing to adjust its complexity to devices of different computational power.
  arXiv  Detail & Related papers  (2022-11-08T17:40:50Z)
• AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation [104.0979785739202]
  Mixture-of-Expert (MoE) models have obtained state-of-the-art performance in Neural Machine Translation (NMT) tasks. Existing MoE models mostly consider a homogeneous design where the same number of experts of the same size are placed uniformly throughout the network. We develop AutoMoE -- a framework for designing heterogeneous MoE's under computational constraints.
  arXiv  Detail & Related papers  (2022-10-14T05:32:17Z)
• MOAT: Alternating Mobile Convolution and Attention Brings Strong Vision Models [40.40784209977589]
  This paper presents MOAT, a family of neural networks that build on top of MObile convolution (i.e., inverted residual blocks) and ATtention. We replace a standard Transformer block with a mobile convolution block, and further reorder it before the self-attention operation. Our conceptually simple MOAT networks are surprisingly effective, achieving 89.1% top-1 accuracy on ImageNet-1K with ImageNet-22K pretraining.
  arXiv  Detail & Related papers  (2022-10-04T18:00:06Z)
• On-Device Training Under 256KB Memory [62.95579393237751]
  We propose an algorithm-system co-design framework to make on-device training possible with only 256KB of memory. Our framework is the first solution to enable tiny on-device training of convolutional neural networks under 256KB and 1MB Flash.
  arXiv  Detail & Related papers  (2022-06-30T17:59:08Z)
• Real-time Monocular Depth Estimation with Sparse Supervision on Mobile [2.5425323889482336]
  In recent years, with the increasing availability of mobile devices, accurate and mobile-friendly depth models have gained importance. We show, with key design choices and studies, even existing architecture can reach highly competitive performance. A version of our model achieves 0.1208 W on DIW with 1M parameters and reaches 44 FPS on a mobile GPU.
  arXiv  Detail & Related papers  (2021-05-25T16:33:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.