Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients

• URL: http://arxiv.org/abs/2603.17809v1
• Date: Wed, 18 Mar 2026 15:03:43 GMT
• Title: Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients
• Abstract summary: Large Vision Language Models (LVLMs) have achieved remarkable success in a range of downstream tasks that require multimodal interaction.<n>Existing LVLM quantization methods measure token sensitivity at the modality level, which fails to capture the complex cross-token interactions.<n>Inspired by axiomatic attribution in mechanistic interpretability, we introduce a fine-grained quantization strategy on Quantization-aware Integrated Gradients (QIG)<n>Our method improves the average accuracy of LLaVA-onevision-7B by 1.60%, reducing the gap to its full-precision counterpart to only 1.33%.
• Score: 36.01708036519368
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision Language Models (LVLMs) have achieved remarkable success in a range of downstream tasks that require multimodal interaction, but their capabilities come with substantial computational and memory overhead, which hinders practical deployment. Among numerous acceleration techniques, post-training quantization is a popular and effective strategy for reducing memory cost and accelerating inference. However, existing LVLM quantization methods typically measure token sensitivity at the modality level, which fails to capture the complex cross-token interactions and falls short in quantitatively measuring the quantization error at the token level. As tokens interact within the model, the distinction between modalities gradually diminishes, suggesting the need for fine-grained calibration. Inspired by axiomatic attribution in mechanistic interpretability, we introduce a fine-grained quantization strategy on Quantization-aware Integrated Gradients (QIG), which leverages integrated gradients to quantitatively evaluate token sensitivity and push the granularity from modality level to token level, reflecting both inter-modality and intra-modality dynamics. Extensive experiments on multiple LVLMs under both W4A8 and W3A16 settings show that our method improves accuracy across models and benchmarks with negligible latency overhead. For example, under 3-bit weight-only quantization, our method improves the average accuracy of LLaVA-onevision-7B by 1.60%, reducing the gap to its full-precision counterpart to only 1.33%. The code is available at https://github.com/ucas-xiang/QIG.

Related papers

• GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs [40.50894372538921]
  Mixed-precision quantizations mitigate memory overhead by allocating expert-wise bit-widths based on their importance.<n>Existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization.<n>We propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations.
  arXiv  Detail & Related papers  (2026-05-21T22:23:38Z)
• MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization [14.071776510862824]
  This work proposes MGVQ, a novel vector quantization framework integrating multi-dimensional sensitivity perception and gradient-Hessian fusion.<n>In 2-bit quantization settings, MGVQ surpasses existing advanced post-training quantization methods significantly, achieving a maximum accuracy improvement of 4.9 points.<n>The proposed method realizes stable and efficient ultra-low-bit VLM quantization, greatly promoting the practical deployment of multimodal large models in resource-limited environments.
  arXiv  Detail & Related papers  (2026-05-20T06:11:25Z)
• LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation [29.676785178178104]
  We present LBLLM, a lightweight binarization framework that achieves W(1+1)A4 quantization through a novel three-stage quantization strategy.<n>LBLLM, trained only using 0.016B tokens with a single GPU, surpasses existing state-of-the-art binarization methods on W2A4 quantization settings.
  arXiv  Detail & Related papers  (2026-04-21T07:25:02Z)
• QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Models [21.01470580488428]
  Vision-language-action (VLA) models unify perception, language, and control for embodied agents.<n>We introduce QuantVLA, a training-free post-training quantization framework.<n>It is the first PTQ approach for VLA systems and the first to successfully quantize a diffusion transformer (DiT) action head.
  arXiv  Detail & Related papers  (2026-02-23T19:55:54Z)
• LSGQuant: Layer-Sensitivity Guided Quantization for One-Step Diffusion Real-World Video Super-Resolution [52.627063566555194]
  We introduce LSGQuant, a layer-sensitivity guided quantizing approach for one-step diffusion-based real-world VSR.<n>Our method incorporates a Dynamic Range Adaptive Quantizer (DRAQ) to fit video token activations.<n>Our method has nearly performance to origin model with full-precision and significantly exceeds existing quantization techniques.
  arXiv  Detail & Related papers  (2026-02-03T06:53:19Z)
• Mixed-Precision Quantization for Language Models: Techniques and Prospects [10.345914140081925]
  Quantization has emerged as an essential compression technique to reduce model size, alleviate memory bottlenecks, and accelerate inference.<n>Mixed-precision quantization offers a promising alternative by selectively allocating precision across layers or within tensors to balance efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-10-19T12:16:40Z)
• RSQ: Learning from Important Tokens Leads to Better Quantized LLMs [65.5558181902098]
  Layer-wise quantization is a key technique for efficiently compressing large models without expensive retraining.<n>We propose RSQ (Rotate, Scale, then Quantize), which applies rotations to the model to mitigate outliers.<n>We demonstrate that RSQ consistently outperforms baseline methods across multiple downstream tasks and three model families.
  arXiv  Detail & Related papers  (2025-03-03T18:46:33Z)
• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [55.73370804397226]
  Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating large language models. We present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats.
  arXiv  Detail & Related papers  (2024-05-09T11:49:05Z)
• Norm Tweaking: High-performance Low-bit Quantization of Large Language Models [21.855106896725598]
  We introduce a technique called norm tweaking, which can be used as a plugin in current PTQ methods to achieve high precision. Our method demonstrates significant improvements in both weight-only quantization and joint quantization of weights and activations. Our simple and effective approach makes it more practical for real-world applications.
  arXiv  Detail & Related papers  (2023-09-06T06:51:15Z)
• Neural Networks with Quantization Constraints [111.42313650830248]
  We present a constrained learning approach to quantization training. We show that the resulting problem is strongly dual and does away with gradient estimations. We demonstrate that the proposed approach exhibits competitive performance in image classification tasks.
  arXiv  Detail & Related papers  (2022-10-27T17:12:48Z)
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition [67.95996816744251]
  State-of-the-art language models (LMs) represented by long-short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming increasingly complex and expensive for practical applications. Current quantization methods are based on uniform precision and fail to account for the varying performance sensitivity at different parts of LMs to quantization errors. Novel mixed precision neural network LM quantization methods are proposed in this paper.
  arXiv  Detail & Related papers  (2021-11-29T12:24:02Z)
• Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning [75.45968495410047]
  Communication of model updates between client nodes and the central aggregating server is a major bottleneck in federated learning. Gradient quantization is an effective way of reducing the number of bits required to communicate each model update. We propose an adaptive quantization strategy called AdaFL that aims to achieve communication efficiency as well as a low error floor.
  arXiv  Detail & Related papers  (2021-02-08T19:14:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.