Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization

• URL: http://arxiv.org/abs/2605.20295v1
• Date: Tue, 19 May 2026 10:48:35 GMT
• Title: Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization
• Abstract summary: Quant.npu is a integer-only fully static quantization framework for NPU-constrained inference.<n>It incorporates learnable quantization parameters and rotation matrices, enabling low-bit activation-weight quantization without runtime quantization parameters re-computation.<n>Quant.npu achieves comparable accuracy to state-of-the-art methods, while reducing inference latency by up to 15.1%.
• Score: 16.937297388706043
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly deployed on mobile devices, where Neural Processing Units (NPUs) necessitate fully static quantization for optimal inference efficiency. However, existing post-training quantization (PTQ) methods predominantly rely on dynamic activation quantization, rendering them incompatible with NPU hardware constraints. To bridge the gap between high-fidelity PTQ and NPU-constrained inference, we propose Quant.npu, a integer-only fully static quantization framework. It incorporates learnable quantization parameters and rotation matrices, enabling low-bit activation-weight quantization without runtime quantization parameters re-computation. Crucially, we identify that initialization and selective optimization of quantization parameters is pivotal for optimization stability, as improper initialization and naive joint optimization induce gradient instability that disrupts the optimization of rotation matrices. To address this, we propose a rotation-and-bit-width-aware initialization tailored to diverse activation profiles and a distribution-aware selective optimization (two-stage quantization pipeline) tailored to rotated and unrotated tensors. Furthermore, we introduce a sensitivity-guided adaptive mixed-precision scheme to balance accuracy with inference efficiency. Extensive experiments on real-world mobile NPUs demonstrate that Quant.npu achieves comparable accuracy to state-of-the-art methods, while reducing inference latency by up to 15.1%.

Related papers

• Stochastic Pauli-path simulator for large-scale quantum optimization [57.179828941706006]
  Pauli-path simulators offer a promising route to large-scale classical simulation of quantum circuits in the low-magic regime.<n>Existing approaches either lack native support for gradient-based optimization or suffer from severe gradient bias.<n>Here we propose the Pauli-path simulator (SPPS), a computational framework for large-scale quantum optimization.
  arXiv  Detail & Related papers  (2026-07-20T10:43:44Z)
• Balancing Expressivity and Learnability in Quantum Kernel Bandit Optimization [50.880384999888044]
  We develop misspecified GP bandit algorithms and derive regret bounds that characterize the trade-off between approximation error and information gain.<n>Our methods outperform full quantum kernels in sample efficiency, while substantially reducing computational overhead.
  arXiv  Detail & Related papers  (2026-07-01T15:38:00Z)
• WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization [40.655670203062805]
  WINDQuant is a reinforcement-learning-based allocation controller for ultra-low-bit LLM quantization.<n>It learns how to assign bit-widths and quantization treatments to fine-grained column chunks under a global storage budget.<n>Experiments on LLaMA models demonstrate that WINDQuant achieves competitive performance in ultra-low-bit settings.
  arXiv  Detail & Related papers  (2026-05-26T07:46:13Z)
• Quantum Approximate Optimization Algorithm with Fixed Number of Parameters [0.0]
  We introduce a novel quantum optimization paradigm: the Fixed--Count Approximate Quantum Optimization Algorithm (FPC-QAOA)<n>It is a scalable variational framework that maintains a constant number of trainable parameters regardless of the number of qubits, Hamiltonian complexity, or circuit depth.<n>We benchmark FPC-QAOA on random MaxCut instances and the Tail Assignment Problem, achieving performance comparable to or better than standard QAOA.
  arXiv  Detail & Related papers  (2025-12-24T14:02:31Z)
• TensorHyper-VQC: A Tensor-Train-Guided Hypernetwork for Robust and Scalable Variational Quantum Computing [50.95799256262098]
  We introduceHyper-VQC, a novel tensor-train (TT)-guided hypernetwork framework for quantum machine learning.<n>Our framework delegates the generation of quantum circuit parameters to a classical TT network, effectively decoupling optimization from quantum hardware.<n>These results positionHyper-VQC as a scalable and noise-resilient framework for advancing practical quantum machine learning on near-term devices.
  arXiv  Detail & Related papers  (2025-08-01T23:37:55Z)
• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• i-QLS: Quantum-supported Algorithm for Least Squares Optimization in Non-Linear Regression [4.737806718785056]
  We propose an iterative quantum-assisted least squares (i-QLS) optimization method.<n>We overcome the scalability and precision limitations of prior quantum least squares approaches.<n> Experiments confirm that i-QLS enables near-term quantum hardware to perform regression tasks with improved precision and scalability.
  arXiv  Detail & Related papers  (2025-05-05T17:02:35Z)
• BoA: Attention-aware Post-training Quantization without Backpropagation [11.096116957844014]
  Post-training quantization is a promising solution for deploying large language models on resource-constrained devices.<n>We introduce a novel backpropagation-free PTQ algorithm that optimize quantized weights by considering inter-layer dependencies.
  arXiv  Detail & Related papers  (2024-06-19T11:53:21Z)
• PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks [4.827161693957252]
  Non-quantized elementwise operations dominate the inference cost of low-precision models. PikeLPN model addresses these issues by applying quantization to both elementwise operations and multiply-accumulate operations.
  arXiv  Detail & Related papers  (2024-03-29T18:23:34Z)
• Squat: Quant Small Language Models on the Edge [45.448118471829474]
  A growing trend has emerged in designing high-quality Small Language Models (SLMs) with a few million parameters.<n>Considering that full parameter training is feasible for SLMs on mobile devices, Quantization-Aware Training (QAT) is employed to improve efficiency.<n>We propose Squat method, an effective QAT framework with deployable quantization for SLMs on mobile devices.
  arXiv  Detail & Related papers  (2024-02-16T16:10:38Z)
• On-Chip Hardware-Aware Quantization for Mixed Precision Neural Networks [52.97107229149988]
  We propose an On-Chip Hardware-Aware Quantization framework, performing hardware-aware mixed-precision quantization on deployed edge devices. For efficiency metrics, we built an On-Chip Quantization Aware pipeline, which allows the quantization process to perceive the actual hardware efficiency of the quantization operator. For accuracy metrics, we propose Mask-Guided Quantization Estimation technology to effectively estimate the accuracy impact of operators in the on-chip scenario.
  arXiv  Detail & Related papers  (2023-09-05T04:39:34Z)
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models [57.27101446992148]
  Large language models (LLMs) have revolutionized natural language processing tasks. Recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM. We introduce an Omnidirectionally calibrated Quantization technique for LLMs, which achieves good performance in diverse quantization settings.
  arXiv  Detail & Related papers  (2023-08-25T02:28:35Z)
• Adaptive pruning-based optimization of parameterized quantum circuits [62.997667081978825]
  Variisy hybrid quantum-classical algorithms are powerful tools to maximize the use of Noisy Intermediate Scale Quantum devices. We propose a strategy for such ansatze used in variational quantum algorithms, which we call "Efficient Circuit Training" (PECT) Instead of optimizing all of the ansatz parameters at once, PECT launches a sequence of variational algorithms.
  arXiv  Detail & Related papers  (2020-10-01T18:14:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.