FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization

• URL: http://arxiv.org/abs/2506.12024v1
• Date: Wed, 21 May 2025 07:42:53 GMT
• Title: FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization
• Authors: Fangxin Liu, Zongwu Wang, JinHong Xia, Junping Zhao, Jian Liu, Haibing Guan, Li Jiang,
• Abstract summary: We propose FlexQuant, a dynamic precision-switching framework to optimize the trade-off between inference speed and accuracy.<n>Our work provides a comprehensive analysis of quantization strategies, introduces a precision requirement model for optimal switching, and implements efficient fine-grained precision management.<n> Experimental results demonstrate that FlexQuant achieves a 1.3x end-to-end speedup across diverse language tasks with negligible accuracy loss.
• Score: 18.041828697950812
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid advancement of large language models (LLMs) has exacerbated the memory bottleneck due to the widening gap between model parameter scaling and hardware capabilities. While post-training quantization (PTQ) techniques effectively reduce memory overhead, existing methods predominantly rely on static quantization strategies, which struggle to adapt to dynamic workloads. To address this, we propose FlexQuant, a dynamic precision-switching framework that optimizes the trade-off between inference speed and accuracy. Leveraging model perplexity entropy and Kullback-Leibler (KL) divergence, FlexQuant enables fine-grained, layer-wise mixed-precision quantization and dynamically adjusts bit-widths during each token generation. Our work provides a comprehensive analysis of quantization strategies, introduces a precision requirement model for optimal switching, and implements efficient fine-grained precision management. Experimental results demonstrate that FlexQuant achieves a 1.3x end-to-end speedup across diverse language tasks with negligible accuracy loss introduced. This framework offers a flexible and adaptive solution for efficient LLM deployment.

Related papers

• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator [5.985414012866983]
  Large language model (LLM) pruning with fixed N:M structured sparsity limits the expressivity of the sparse model.<n>We present a flexible layer-wise outlier-density-aware N:M sparsity (FLOW) selection method.<n>We then introduce a flexible, low-overhead digital compute-in-memory architecture (FlexCiM)
  arXiv  Detail & Related papers  (2025-04-19T17:47:01Z)
• Q-MambaIR: Accurate Quantized Mamba for Efficient Image Restoration [34.43633070396096]
  State-Space Models (SSMs) have attracted considerable attention in Image Restoration (IR)<n>Q-MambaIR is an accurate, efficient, and flexible Quantized Mamba for IR tasks.
  arXiv  Detail & Related papers  (2025-03-27T20:34:11Z)
• QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models [3.093903491123962]
  Large language models (LLMs) have significantly advanced various natural language processing (NLP) tasks.<n> structured pruning is an effective approach to reducing model size, but it often results in significant accuracy degradation.<n>We introduce quantization into the structured pruning framework to reduce memory consumption during both fine-tuning and inference.<n>We propose QPruner, a novel framework that employs structured pruning to reduce model size, followed by a layer-wise mixed-precision quantization scheme.
  arXiv  Detail & Related papers  (2024-12-16T10:14:01Z)
• AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment [13.977849745488339]
  AmoebaLLM is a novel framework designed to enable the instant derivation of large language models of arbitrary shapes. AmoebaLLM significantly facilitates rapid deployment tailored to various platforms and applications.
  arXiv  Detail & Related papers  (2024-11-15T22:02:28Z)
• Progressive Mixed-Precision Decoding for Efficient LLM Inference [49.05448842542558]
  We introduce Progressive Mixed-Precision Decoding (PMPD) to address the memory-boundedness of decoding.<n>PMPD achieves 1.4$-$12.2$times$ speedup in matrix-vector multiplications over fp16 models.<n>Our approach delivers a throughput gain of 3.8$-$8.0$times$ over fp16 models and up to 1.54$times$ over uniform quantization approaches.
  arXiv  Detail & Related papers  (2024-10-17T11:46:33Z)
• LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference [6.185573921868495]
  We present textbfLLMEasyQuant, a system-aware quantization framework for large language models (LLMs)<n>It is designed for efficient, low-bit inference of LLMs on single-node multi-GPU, multi-node, and edge hardware.
  arXiv  Detail & Related papers  (2024-06-28T04:56:53Z)
• Flextron: Many-in-One Flexible Large Language Model [85.93260172698398]
  We introduce Flextron, a network architecture and post-training model optimization framework supporting flexible model deployment. We present a sample-efficient training method and associated routing algorithms for transforming an existing trained LLM into a Flextron model. We demonstrate superior performance over multiple end-to-end trained variants and other state-of-the-art elastic networks, all with a single pretraining run that consumes a mere 7.63% tokens compared to original pretraining.
  arXiv  Detail & Related papers  (2024-06-11T01:16:10Z)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models [63.118592279833656]
  Post-training quantization (PTQ) is an effective technique for compressing large language models (LLMs)<n>We propose SliM-LLM, a salience-driven mixed-precision quantization framework that allocates bit-widths at the group-wise.<n> Experiments show that SliM-LLM achieves superior performance across various LLMs at low bit-widths.
  arXiv  Detail & Related papers  (2024-05-23T16:21:48Z)
• QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources [35.16907522675046]
  Large Language Models (LLMs) have showcased remarkable impacts across a wide spectrum of natural language processing tasks.<n>Fine-tuning these pretrained models on downstream datasets provides significant performance gains.<n>This process typically requires a large number of expensive, high-end GPU.<n>We propose QFT, a Quantized Full- parameter Tuning framework that quantizes and stores all training states.
  arXiv  Detail & Related papers  (2023-10-11T02:47:40Z)
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models [57.27101446992148]
  Large language models (LLMs) have revolutionized natural language processing tasks. Recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM. We introduce an Omnidirectionally calibrated Quantization technique for LLMs, which achieves good performance in diverse quantization settings.
  arXiv  Detail & Related papers  (2023-08-25T02:28:35Z)
• Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures [68.91874045918112]
  adapter-ALBERT is an efficient model optimization for maximal data reuse across different tasks. We demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator.
  arXiv  Detail & Related papers  (2023-03-25T14:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.