SAB-LVLM: Significance-Aware Binarization for Large Vision-Language Models
Abstract Overview
This paper studies post-training weight binarization for large vision-language models (LVLMs) under an approximately 1-bit compression regime, targeting the memory and latency burden of multimodal deployment. The core idea is to make binarization significance-aware rather than treating all weights equally: the method estimates textual and visual weight sensitivity with Hessian-based statistics from calibration data, builds a spatial significance map to separate predominantly unimodal from cross-modal weights, and integrates these signals into a significance-aware binarization map. The resulting map is used to reweight the binarization objective, and the low-bit weights are optimized with an alternating significance-weighted update scheme. Experiments on Qwen2.5-VL and InternVL3.5 families across MMStar, DocVQA, TextVQA, Video-MME, and VSI-Bench show that this approach preserves multimodal capability better than prior 1-bit post-training binarization baselines.
Novelty
The paper's main novelty is a significance-aware binarization framework designed specifically for LVLMs, where weight importance is modeled jointly across layers and across text and vision modalities. Its distinctive components are the spatial significance map, modality-guided significance integration, and an alternating update procedure that injects these significance weights directly into the binarization objective.
Results
Across multiple Qwen2.5-VL and InternVL3.5 models, SAB-LVLM consistently outperforms prior 1-bit baselines such as PB-LLM, BiLLM, and ARB-LLM while keeping roughly the same ~1.07-bit storage budget. For example, on Qwen2.5-VL-32B-Instruct it reaches 54.77 on MMStar versus 48.48 for ARB-LLM at the same 10.28 GB memory, and on Qwen2.5-VL-72B-Instruct it achieves 64.93 on Video-MME versus 61.15 for ARB-LLM. Ablations further show that adaptive modality-guided integration and the full spatial significance map are important contributors to the gains.
Key Points
- SAB-LVLM estimates weight significance separately for text and image inputs using Hessian-based sensitivity, then distinguishes unimodal and multimodal activation patterns through a spatial significance map.
- The method reweights the binarization loss with a significance-aware map and optimizes binary weights via alternating significance-weighted updates, rather than minimizing global quantization error alone.
- Empirically, SAB-LVLM improves 1-bit LVLM binarization across five multimodal benchmarks and several Qwen2.5-VL and InternVL3.5 model sizes, while maintaining approximately 1-bit compression efficiency.