HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation

• URL: http://arxiv.org/abs/2605.17093v1
• Date: Sat, 16 May 2026 17:33:24 GMT
• Title: HEED: Density-Weighted Residual Alignment for Hybrid Vision-Language Model Distillation
• Abstract summary: We distill Qwen3-VL-8B-Instruct into a 3:1 Mamba-2/attention hybrid.<n>Student model stays within 2 points of the teacher across visual reasoning benchmarks like MMStar, MMBench, and MMMU-Pro.<n>The student can still understand the scene but loses the fine-grained text needed to answer.<n>After standard post-training, the student reaches teacher-level performance on the 10-benchmark average with a 4.12$times$ throughput and a 68% memory saving at 128k context.
• Score: 5.501291336853232
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Distilling vision-language models into faster hybrid architectures, such as 3:1 Mamba-2/attention mixes, is now standard practice for making inference efficient. Aggregate benchmarks suggest that this works but they hide selective failures. When we distill Qwen3-VL-8B-Instruct into a 3:1 Mamba-2/attention hybrid, student model stays within 2 points of the teacher across visual reasoning benchmarks like MMStar, MMBench, and MMMU-Pro, while dropping 13 points on optical-character-recognition and document tasks. The student can still understand the scene but loses the fine-grained text needed to answer. We localize much of the failure to a specific kind of position. In a high-resolution image, most patches are sky, wall, or smooth texture, while a small fraction carries text, edges, object boundaries, or other local details. In a token-level diagnostic, the top 10% highest-density patches have 3.6$\times$ larger residual drift than the bottom 10% lowest-density patches and 3.5$\times$ larger teacher-masking answer contribution. Uniform weighting devotes many loss terms to low-information background patches, whereas sparse answer-bearing patches receive no special protection. The required intervention is minimal: we replace uniform residual alignment with density-weighted residual alignment, using patch self-dissimilarity as a training-free proxy for position importance. We call this HEED. Compared with normal end-to-end distillation, HEED increases performance by 8.7 points on OCRBench v2 and 5.13 points on a 10-benchmark average. The gain is realized on different teacher models and hybrid architectures. After standard post-training, the student reaches teacher-level performance on the 10-benchmark average with a 4.12$\times$ throughput and a 68% memory saving at 128k context, with no additional parameters and no inference-time cost.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Cross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification [10.122947222129108]
  Self-supervised vision foundation models such as DINOv2 provide strong features but are too large for field deployment.<n>We study cross-architecture knowledge distillation from a fine-tuned DINOv2 teacher to a compact bidirectional Visual State Space Model.<n>We identify and fix two training-stability problems that prevent the from-scratch SSM student from learning on limited data.
  arXiv  Detail & Related papers  (2026-08-27T08:02:05Z)
• DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance [12.934094486594633]
  We present DistillPath-KS16, which starts from the existing 22M kaiko ViT-S/16 encoder and improves it by distilling from released pathology encoders used as frozen teachers.<n>The recipe reads only the teachers' final class and patch tokens and trains on 6,000 public slides, needing neither their DINO nor iBOT pretraining heads nor a billion-tile corpus.
  arXiv  Detail & Related papers  (2026-08-18T15:06:35Z)
• Visual-OPSD: Cross-Modal On-Policy Self-Distillation for Efficient Unified Multimodal Reasoning [30.851590309402436]
  Unified multimodal models (UMMs) interleave generated ''visual thoughts'' (VTs) with text reasoning to improve spatial tasks.<n>This incurs roughly an order-of-magnitude inference cost from multi-step diffusion.<n>We propose Visual On-Policy Self-Distillation(Visual-OPSD) to solve this problem.
  arXiv  Detail & Related papers  (2026-06-17T11:59:15Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation [49.98710755440242]
  Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities.<n>We reveal a critical inefficiency in this approach: up to 69% of the prompts in standard chart / document reasoning datasets are effectively zero-delta.<n>We propose a staged synthesis pipeline that repurposes existing datasets as seeds, actively targeting student failure modes to produce better prompts.
  arXiv  Detail & Related papers  (2026-05-15T02:04:12Z)
• TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM [18.69783590125584]
  Diffusion large language models (dLLMs) offer a promising paradigm for parallel text generation.<n>They face an accuracy-parallelism trade-off, where increasing tokens per forward (TPF) often degrades generation quality.<n>Existing acceleration methods often gain speed at the cost of accuracy.<n>We propose TAD, a Temporal-Aware trajectory self-Distillation framework.
  arXiv  Detail & Related papers  (2026-05-10T13:38:53Z)
• Training a Student Expert via Semi-Supervised Foundation Model Distillation [14.605642634292655]
  We introduce a semi-supervised knowledge distillation framework that compresses pre-trained vision foundation models into compact experts.<n>The framework unfolds in three stages: (1) domain adaptation of the VFM(s) via self-training with contrastive calibration, (2) knowledge transfer through a unified multi-objective loss, and (3) student refinement to mitigate residual pseudo-label bias.
  arXiv  Detail & Related papers  (2026-04-04T19:45:25Z)
• Mixture-of-Depths Attention [65.80640499676542]
  Scaling depth is a key driver for large language models (LLMs)<n>We introduce mixture-of-depths attention (MoDA)<n>MoDA allows each attention head to attend to sequence KV pairs at the current layer and depth KV pairs from preceding layers.
  arXiv  Detail & Related papers  (2026-03-16T17:59:55Z)
• A Boundary-Metric Evaluation Protocol for Whiteboard Stroke Segmentation Under Extreme Imbalance [0.0]
  Boundary-aware metrics and a thin-subset equity analysis change how loss functions rank and exposes hidden trade-offs.<n>Five losses are trained three times each on a DeepLabV3-MobileNetV3 model and evaluated on 12 held-out images.<n>Overlap-based losses improve F1 by more than 20 points over cross-entropy.
  arXiv  Detail & Related papers  (2026-02-26T05:16:18Z)
• Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference [54.467557491325046]
  We propose Pre-hoc Sparsity (PrHS), which selects KV entries before attention scoring and provides explicit accuracy control.<n>PrHS reduces retrieval overhead by over 90%, achieving 3x higher retrieval sparsity than HShare at matched or better accuracy.<n>It incurs under 1% average degradation on LongBench, lowers attention FLOPs by about 15% versus prior baselines, and yields a 9.9x speedup in attention-operator latency and 2.8x higher throughput.
  arXiv  Detail & Related papers  (2026-02-09T07:05:23Z)
• CAST: Contrastive Adaptation and Distillation for Semi-Supervised Instance Segmentation [14.605642634292655]
  We introduce CAST, a semi-supervised knowledge distillation (SSKD) framework that compresses pre-trained vision foundation models (VFM) into compact experts using limited labeled and abundant unlabeled data.<n>Cast unfolds in three stages: (1) domain adaptation of the VFM(s) via self-training with contrastive calibration, (2) knowledge transfer through a unified multi-objective loss, and (3) student refinement to mitigate residual pseudo-label bias.
  arXiv  Detail & Related papers  (2025-05-28T02:45:42Z)
• Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization [47.38380084735716]
  vision-supervised models (VLMs), pre-trained on massive image-text pairs, have demonstrated remarkable zero-/few-shot performance.<n>Knowledge distillation (KD) offers a natural framework for transferring VLM capabilities, but it suffers from gradient conflicts between supervised and distillation losses.<n>We propose Dual-Head Optimization (DHO), which introduces dual prediction heads for each distinct signal.
  arXiv  Detail & Related papers  (2025-05-12T15:39:51Z)
• A Quadratic Synchronization Rule for Distributed Deep Learning [66.68264684667562]
  This work proposes a theory-grounded method for determining $H$, named the Quadratic Synchronization Rule (QSR) Experiments on ResNet and ViT show that local gradient methods with QSR consistently improve the test accuracy over other synchronization strategies.
  arXiv  Detail & Related papers  (2023-10-22T21:38:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.