Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models

• URL: http://arxiv.org/abs/2605.19929v1
• Date: Tue, 19 May 2026 14:49:57 GMT
• Title: Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
• Abstract summary: Low-bit post-training quantization (PTQ) is a pivotal technique for deploying Vision-Language Models (VLMs) on resource-constrained devices.<n>Existing PTQ methods often degrade VLMs' accuracy due to the heterogeneous activation distributions of text and vision modalities during quantization.<n>We propose SplitQ, a channel-Splitting-driven post-training Quantization framework.
• Score: 36.14224700486847
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-bit post-training quantization (PTQ) is a pivotal technique for deploying Vision-Language Models (VLMs) on resource-constrained devices. However, existing PTQ methods often degrade VLMs' accuracy due to the heterogeneous activation distributions of text and vision modalities during quantization. We find that this cross-modal heterogeneity is distributed unevenly across channels: a small subset of channels contains most modality-specific outliers, and these outliers typically reside in different channels for each modality. Motivated by this, we propose SplitQ, a channel-Splitting-driven post-training Quantization framework. At its core, SplitQ introduces a novel Modality-specific Outlier Channel Decoupling (MOCD) module that effectively isolates salient modality-specific outlier channels with minimal overhead. To further address the remaining cross-modal distribution discrepancies, we design an Adaptive Cross-Modal Calibration (ACC) module that employs dual lightweight learnable branches to dynamically mitigate modality-induced quantization errors. Extensive experiments on popular VLMs demonstrate that SplitQ significantly outperforms existing approaches across 6 popular multi-modal datasets under all evaluated quantization settings, including W4A8, W4A4, W3A3, and W3A2. Notably, SplitQ preserves 93.5% of FP16 performance under the challenging W3A3 setting (69.5 vs. 74.3), pushing the efficiency frontier for deploying advanced VLMs. Our code is available at https://github.com/EMVision-NK/SplitQ

Related papers

• MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models [57.87791117741788]
  We propose a modality-aware PTQ framework to preserve cross-modal morphology and mitigate outlier loss.<n>Distribution-Aware Bias Compensation (DABC) selectively absorbs long-tailed outliers into channel-wise biases.<n>We also propose Morphology-Directed Quantization Function Optimization (MDQFO) to co-optimize the quantization grid with the bias mask.
  arXiv  Detail & Related papers  (2026-06-03T02:05:10Z)
• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond [50.440302567029654]
  Multi-modal intelligence has made the Key-Value cache a dominant memory bottleneck for efficient deployment.<n>In this work, we revisit the inherent limitations of the per-channel quantization paradigm.<n>We propose OScaR, an accurate and lightweight KV cache compression framework for X-LLMs.
  arXiv  Detail & Related papers  (2026-05-19T10:53:03Z)
• PTQ4ARVG: Post-Training Quantization for AutoRegressive Visual Generation Models [22.35209793690791]
  Quantization is commonly employed in neural networks to reduce model size and computational latency.<n>Applying quantization to ARVG remains largely underexplored.<n>We propose PTQ4ARVG, a training-free post-training quantization framework.
  arXiv  Detail & Related papers  (2026-01-29T04:00:00Z)
• ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs [4.431548809730958]
  ARCQuant is a framework that boosts NVFP4 performance via Augmented Residual Channels.<n>We show that ARCQuant achieves state-of-the-art accuracy, comparable to full-precision baselines in perplexity and downstream tasks.
  arXiv  Detail & Related papers  (2026-01-12T12:27:22Z)
• BasicAVSR: Arbitrary-Scale Video Super-Resolution via Image Priors and Enhanced Motion Compensation [70.27358326228399]
  We propose a BasicAVSR for Arbitrary-scale video super-resolution (AVSR)<n>AVSR aims to enhance the resolution of video frames, potentially various scaling factors.<n>We show that BasicAVSR significantly outperforms existing methods in terms of super-resolution quality, generalization ability, and inference speed.
  arXiv  Detail & Related papers  (2025-10-30T05:08:45Z)
• TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models [49.65286242048452]
  We propose a novel method dubbed Timestep-Channel Adaptive Quantization for Diffusion Models (TCAQ-DM)<n>The proposed method substantially outperforms the state-of-the-art approaches in most cases.
  arXiv  Detail & Related papers  (2024-12-21T16:57:54Z)
• Instance-Aware Group Quantization for Vision Transformers [20.105148326987646]
  Post-training quantization (PTQ) is an efficient model compression technique that quantizes a pretrained full-precision model. PTQ methods for convolutional neural networks (CNNs) provide quantization results comparable to full-precision counterparts. We introduce instance-aware group quantization for ViTs (IGQ-ViT)
  arXiv  Detail & Related papers  (2024-04-01T05:12:30Z)
• Sparsely Multimodal Data Fusion [0.0]
  This paper presents a comparative study of three multimodal embedding techniques, Modal Channel Attention (MCA), Zorro, and Everything at Once (EAO)<n>MCA introduces fusion embeddings for all combinations of input modalities and uses attention masking to create distinct attention channels.<n>MCA achieves superior performance by maintaining robust uniformity across unimodal and fusion embeddings.
  arXiv  Detail & Related papers  (2024-03-29T16:49:40Z)
• Squeezeformer: An Efficient Transformer for Automatic Speech Recognition [99.349598600887]
  Conformer is the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture. We propose the Squeezeformer model, which consistently outperforms the state-of-the-art ASR models under the same training schemes.
  arXiv  Detail & Related papers  (2022-06-02T06:06:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.