Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

• URL: http://arxiv.org/abs/2601.22709v2
• Date: Mon, 02 Feb 2026 06:39:48 GMT
• Title: Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
• Abstract summary: Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss.<n>We propose GRACE, a framework unifying knowledge distillation and QAT under the Information Bottleneck principle.<n>Treating the teacher as a proxy for task-relevant information, we introduce confidence-gated decoupled distillation to filter unreliable supervision.
• Score: 22.280338887759623
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) achieve strong multimodal performance but are costly to deploy, and post-training quantization often causes significant accuracy loss. Despite its potential, quantization-aware training for VLMs remains underexplored. We propose GRACE, a framework unifying knowledge distillation and QAT under the Information Bottleneck principle: quantization constrains information capacity while distillation guides what to preserve within this budget. Treating the teacher as a proxy for task-relevant information, we introduce confidence-gated decoupled distillation to filter unreliable supervision, relational centered kernel alignment to transfer visual token structures, and an adaptive controller via Lagrangian relaxation to balance fidelity against capacity constraints. Across extensive benchmarks on LLaVA and Qwen families, our INT4 models consistently outperform FP16 baselines (e.g., LLaVA-1.5-7B: 70.1 vs. 66.8 on SQA; Qwen2-VL-2B: 76.9 vs. 72.6 on MMBench), nearly matching teacher performance. Using real INT4 kernel, we achieve 3$\times$ throughput with 54% memory reduction. This principled framework significantly outperforms existing quantization methods, making GRACE a compelling solution for resource-constrained deployment.

Related papers

• Towards Compact Unified Multimodal Tracking: Synergizing Knowledge Distillation with Structural Pruning [80.73561460934809]
  multimodal object tracking has achieved remarkable robustness by leveraging complementary sensor data.<n>We identify the prediction head as a critical but often overlooked efficiency bottleneck.<n>By strategically streamlining the decoder architecture, we unlock the potential for real-time inference.
  arXiv  Detail & Related papers  (2026-08-02T20:41:33Z)
• Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation [10.4712452422576]
  Quantization-aware distillation (QAD) helps recover accuracy lost under low bit quantization.<n>We show that KL-only QAD can reduce layerwise representational similarity relative to the BF16 teacher.<n>We propose textbfCKA-QAD, a CKA-guided representational alignment method for NVFP4 QAD and low bit LLM accuracy recovery.
  arXiv  Detail & Related papers  (2026-06-04T04:03:42Z)
• ReAD: Reinforcement-Guided Capability Distillation for Large Language Models [69.00808591939524]
  We study capability distillation under a fixed token budget and identify two consistent patterns.<n>We propose ReAD, a Reinforcement-guided cApability Distillation framework.
  arXiv  Detail & Related papers  (2026-05-11T22:17:51Z)
• DurableUn: Quantization-Induced Recovery Attacks in Machine Unlearning [8.466251176705423]
  We show that INT4 quantization restores forgotten content even when models pass compliance audits at bfloat16 (BF16)<n>We conduct the first systematic study of unlearning robustness under adapter-space INT4 quantization in the NFLoRA regime.<n>We call for Q-INT4 to be adopted as a standard evaluation metric alongside FA and RA.
  arXiv  Detail & Related papers  (2026-05-04T03:54:14Z)
• RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference [1.1100764382749708]
  RAMP (Reinforcement Adaptive Mixed Precision) learns per layer bit width assignments to minimize perplexity under a global bit budget.<n>On Llama 2 7B, RAMP achieves 5.54 perplexity at 3.68GB (3.65 effective bits), outperforming uniform 4 bit AWQ (5.60 at 3.90 GB) and GPTQ by 6% in size and 1% to3% in quality.
  arXiv  Detail & Related papers  (2026-03-18T16:16:28Z)
• SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learning [13.174004826305255]
  Multi-stage ML inference pipelines are difficult to autoscale due to heterogeneous resources, cross-stage coupling, and dynamic bottleneck migration.<n>We present SAIR, an autoscaling framework that uses an LLM as an in-context reinforcement learning controller.<n>SAIR achieves the best or tied-best P99 latency and effective resource cost among deployed baselines, improving P99 by up to 50% and reducing effective cost by up to 97%.
  arXiv  Detail & Related papers  (2026-01-29T23:08:15Z)
• What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study [59.44848132298657]
  Post-training quantization (PTQ) usually comes with the cost of large accuracy drops, especially for reasoning tasks under low-bit settings.<n>In this study, we present a systematic empirical study of quantization-aware training (QAT) for reasoning models.
  arXiv  Detail & Related papers  (2026-01-21T11:22:29Z)
• A Fano-Style Accuracy Upper Bound for LLM Single-Pass Reasoning in Multi-Hop QA [65.38186593873313]
  Multi-Hop Question Answering (MHQA) requires integrating dispersed, interdependent evidence through sequential reasoning under noise.<n>We introduce a proof-of-concept multi-call framework for MHQA, InfoQA.<n>We construct a stringent and noise-rich benchmark to validate our theory and framework.
  arXiv  Detail & Related papers  (2025-09-25T14:11:57Z)
• QUADS: QUAntized Distillation Framework for Efficient Speech Language Understanding [0.0]
  Existing methods apply distillation and quantization separately, leading to suboptimal compression.<n>We propose QUADS, a unified framework that optimize both through multi-stage training with a pre-tuned model.<n>QUADS achieves 71.13% accuracy on SLURP and 99.20% on FSC, with only minor degradations of up to 5.56% compared to state-of-the-art models.
  arXiv  Detail & Related papers  (2025-05-19T16:09:51Z)
• Optimizing Large Language Model Training Using FP4 Quantization [73.55459961002371]
  Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce costs.<n>This work introduces the first FP4 training framework for large language models (LLMs)
  arXiv  Detail & Related papers  (2025-01-28T18:04:50Z)
• P4Q: Learning to Prompt for Quantization in Visual-language Models [38.87018242616165]
  We propose a method that balances fine-tuning and quantization named Prompt for Quantization'' (P4Q) Our method can effectively reduce the gap between image features and text features caused by low-bit quantization. Our 8-bit P4Q can theoretically compress the CLIP-ViT/B-32 by 4 $times$ while achieving 66.94% Top-1 accuracy.
  arXiv  Detail & Related papers  (2024-09-26T08:31:27Z)
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving [52.31791050376249]
  Quantization can accelerate large language model (LLM) inference.<n>We introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache.<n>QServe improves the maximum achievable serving of Llama-3-8B by 1.2x on A100, 1.4x on L40S; and Qwen-721.5B by 2.4x on A100, 3.5x on L40S.
  arXiv  Detail & Related papers  (2024-05-07T17:59:30Z)
• Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models [5.69541128149828]
  Large generative models such as large language models (LLMs) and diffusion models have revolutionized the fields of NLP and computer vision respectively. In this study, we propose a light-weight quantization aware fine tuning technique using knowledge distillation (KD-QAT) to improve the performance of 4-bit weight quantized LLMs. We show that ov-freeze results in near floating point precision performance, i.e., less than 0.7% loss of accuracy on Commonsense Reasoning benchmarks.
  arXiv  Detail & Related papers  (2024-03-26T23:51:44Z)
• ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats [25.543571445739936]
  This study explores the viability of floating-point (FP) quantization for large language models (LLMs) For LLMs, FP8 activation consistently outshines its integer (INT8) equivalent, with the performance edge becoming more noticeable in models possessing parameters beyond one billion. For weight quantization, our findings indicate that FP4 exhibits comparable, if not superior, performance to INT4, simplifying deployment on FP-supported hardware like H100.
  arXiv  Detail & Related papers  (2023-07-19T06:58:03Z)
• Q-DETR: An Efficient Low-Bit Quantized Detection Transformer [50.00784028552792]
  We find that the bottlenecks of Q-DETR come from the query information distortion through our empirical analyses. We formulate our DRD as a bi-level optimization problem, which can be derived by generalizing the information bottleneck (IB) principle to the learning of Q-DETR. We introduce a new foreground-aware query matching scheme to effectively transfer the teacher information to distillation-desired features to minimize the conditional information entropy.
  arXiv  Detail & Related papers  (2023-04-01T08:05:14Z)
• Generalized Focal Loss V2: Learning Reliable Localization Quality Estimation for Dense Object Detection [78.11775981796367]
  GFLV2 (ResNet-101) achieves 46.2 AP at 14.6 FPS, surpassing the previous state-of-the-art ATSS baseline (43.6 AP at 14.6 FPS) by absolute 2.6 AP on COCO tt test-dev. Code will be available at https://github.com/implus/GFocalV2.
  arXiv  Detail & Related papers  (2020-11-25T17:06:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.