EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens

• URL: http://arxiv.org/abs/2511.21106v1
• Date: Wed, 26 Nov 2025 06:45:59 GMT
• Title: EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens
• Abstract summary: Efficient Multimodal Large Language Models (MLLMs) compress vision tokens to reduce resource consumption, but the loss of visual information can degrade comprehension capabilities.<n>Some priors introduce Knowledge Distillation to enhance student models, but they overlook the fundamental differences in fine-grained vision comprehension.<n>We propose EM-KD, a novel paradigm that enhances the Efficient MLLMs with Knowledge Distillation.
• Score: 37.11253070112327
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Efficient Multimodal Large Language Models (MLLMs) compress vision tokens to reduce resource consumption, but the loss of visual information can degrade comprehension capabilities. Although some priors introduce Knowledge Distillation to enhance student models, they overlook the fundamental differences in fine-grained vision comprehension caused by unbalanced vision tokens between the efficient student and vanilla teacher. In this paper, we propose EM-KD, a novel paradigm that enhances the Efficient MLLMs with Knowledge Distillation. To overcome the challenge of unbalanced vision tokens, we first calculate the Manhattan distance between the vision logits of teacher and student, and then align them in the spatial dimension with the Hungarian matching algorithm. After alignment, EM-KD introduces two distillation strategies: 1) Vision-Language Affinity Distillation (VLAD) and 2) Vision Semantic Distillation (VSD). Specifically, VLAD calculates the affinity matrix between text tokens and aligned vision tokens, and minimizes the smooth L1 distance of the student and the teacher affinity matrices. Considering the semantic richness of vision logits in the final layer, VSD employs the reverse KL divergence to measure the discrete probability distributions of the aligned vision logits over the vocabulary space. Comprehensive evaluation on diverse benchmarks demonstrates that EM-KD trained model outperforms prior Efficient MLLMs on both accuracy and efficiency with a large margin, validating its effectiveness. Compared with previous distillation methods, which are equipped with our proposed vision token matching strategy for fair comparison, EM-KD also achieves better performance.

Related papers

• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models [8.039490357019801]
  Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in visual understanding and reasoning.<n>They also impose significant computational burdens due to long visual sequence inputs.<n>Recent works address this issue by pruning unimportant visual tokens, achieving substantial computational reduction.
  arXiv  Detail & Related papers  (2026-04-27T01:56:59Z)
• Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch [2.752817022620644]
  Large language models (LLMs) achieve state-of-the-art (SOTA) performance across language tasks, but are costly to deploy due to their size and resource demands.<n>Knowledge Distillation (KD) addresses this by training smaller Student models to mimic larger Teacher models, improving efficiency without significant performance loss.
  arXiv  Detail & Related papers  (2026-03-23T14:53:55Z)
• Attention Guided Alignment in Efficient Vision-Language Models [56.20286899428444]
  Large Vision-Language Models (VLMs) rely on effective multimodal alignment between pre-trained vision encoders and Large Language Models (LLMs)<n>This paper presents a comprehensive analysis of attention patterns in efficient VLMs.<n>We introduce Attention-Guided Efficient Vision-Language Models (AGE-VLM), a novel framework that enhances visual grounding through interleaved cross-attention layers.
  arXiv  Detail & Related papers  (2025-11-21T21:36:48Z)
• Learning Compact Vision Tokens for Efficient Large Multimodal Models [11.212952256422609]
  We propose a method to learn compact vision tokens for short vision token sequence.<n>We also introduce a Multi-Block Token Fusion (MBTF) module to supplement multi-granularity features for the reduced token sequence.<n>Our method achieves comparable or even superior performance to the baseline on 8 popular vision-language benchmarks.
  arXiv  Detail & Related papers  (2025-06-08T13:36:06Z)
• VladVA: Discriminative Fine-tuning of LVLMs [67.14293827774827]
  Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the de facto approach for discriminative vision-language representation learning.<n>We propose to combine "the best of both worlds": a new training approach for discriminative fine-tuning of LVLMs.
  arXiv  Detail & Related papers  (2024-12-05T17:54:27Z)
• FoPru: Focal Pruning for Efficient Large Vision-Language Models [11.36025001578531]
  We propose Focal Pruning (FoPru), a training-free method that prunes visual tokens based on the attention-based token significance derived from the vision encoder. Our method can prune a large number of redundant tokens while maintaining high accuracy, leading to significant improvements in inference efficiency.
  arXiv  Detail & Related papers  (2024-11-21T14:22:38Z)
• VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation [66.00245701441547]
  We introduce a novel approach to reduce vision compute by leveraging redundant vision tokens "skipping layers" rather than decreasing the number of vision tokens. Our method, VideoLLM-MoD, is inspired by mixture-of-depths LLMs and addresses the challenge of numerous vision tokens in long-term or streaming video.
  arXiv  Detail & Related papers  (2024-08-29T17:21:58Z)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning [67.0609518552321]
  We propose to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner.
  arXiv  Detail & Related papers  (2023-12-05T07:29:14Z)
• Planting a SEED of Vision in Large Language Model [73.17530130368053]
  We present SEED, an elaborate image tokenizer that empowers Large Language Models (LLMs) with the ability to SEE and Draw at the same time. This version of SEED was trained in 5.7 days using only 64 V100 GPUs and 5M publicly available image-text pairs.
  arXiv  Detail & Related papers  (2023-07-16T13:41:39Z)
• Distilling Large Vision-Language Model with Out-of-Distribution Generalizability [43.984177729641615]
  This paper investigates the distillation of visual representations in large teacher vision-language models into lightweight student models. We propose several metrics and conduct extensive experiments to investigate their techniques. The results demonstrate significant improvements in zero-shot and few-shot student performance on open-vocabulary out-of-distribution classification.
  arXiv  Detail & Related papers  (2023-07-06T17:05:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.