Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

• URL: http://arxiv.org/abs/2603.21426v1
• Date: Sun, 22 Mar 2026 22:33:25 GMT
• Title: Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
• Abstract summary: Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance.<n>We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that modulates how much the student relies on teacher guidance.
• Score: 26.06143154557816
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is challenging, as some samples may be noisy while others are subject to teacher uncertainty. This motivates the need for adaptively balancing data and teacher supervision. We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that adaptively modulates how much the student relies on teacher guidance. Specifically, we formulate teacher--student learning from a unified Bayesian perspective and interpret teacher supervision as a Gibbs prior over student activations. This yields a closed-form, uncertainty-aware weighting mechanism and supports arbitrary distillation objectives and their combinations. Extensive experiments on multimodal VQA benchmarks demonstrate that distilling student Vision-Language Models from a large teacher VLM consistently improves performance. The results show that Beta-KD outperforms existing knowledge distillation methods. The code is available at https://github.com/Jingchensun/beta-kd.

Related papers

• Multi-Teacher Knowledge Distillation via Teacher-Informed Mixture Priors [6.218677961564673]
  We introduce textitMulti-Teacher Bayesian Knowledge Distillation (MT-BKD), where a distilled student model learns from multiple teachers within the Bayesian framework.<n>An entropy-based weighting mechanism adaptively adjusts each teacher's influence, allowing the student to combine multiple sources of expertise effectively.<n>We validate MT-BKD on both synthetic and real-world tasks, including protein subcellular location prediction and image classification.
  arXiv  Detail & Related papers  (2026-05-27T05:03:24Z)
• Curriculum Learning-Guided Progressive Distillation in Large Language Models [18.961191510804134]
  Existing distillation approaches often overlook two critical factors: the learning order of training data and the capacity mismatch between teacher and student models.<n>We propose Curriculum Learning-Guided Progressive Distillation (CLPD), a unified framework that accounts for both factors by aligning data difficulty with teacher strength.<n>Our framework is modular and can be integrated into standard distillation algorithms with minimal overhead.
  arXiv  Detail & Related papers  (2026-05-11T21:37:20Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Improving Knowledge Distillation with Teacher's Explanation [14.935696904019146]
  We introduce a novel Knowledge Explaining Distillation (KED) framework. KED allows the student to learn not only from the teacher's predictions but also from the teacher's explanations. Our experiments over a variety of datasets show that KED students can substantially outperform KD students of similar complexity.
  arXiv  Detail & Related papers  (2023-10-04T04:18:01Z)
• Faithful Knowledge Distillation [75.59907631395849]
  We focus on two crucial questions with regard to a teacher-student pair: (i) do the teacher and student disagree at points close to correctly classified dataset examples, and (ii) is the distilled student as confident as the teacher around dataset examples? These are critical questions when considering the deployment of a smaller student network trained from a robust teacher within a safety-critical setting.
  arXiv  Detail & Related papers  (2023-06-07T13:41:55Z)
• Switchable Online Knowledge Distillation [68.2673580932132]
  Online Knowledge Distillation (OKD) improves involved models by reciprocally exploiting the difference between teacher and student. We propose Switchable Online Knowledge Distillation (SwitOKD) to answer these questions.
  arXiv  Detail & Related papers  (2022-09-12T03:03:40Z)
• Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation [70.92135839545314]
  We propose the dynamic prior knowledge (DPK), which integrates part of teacher's features as the prior knowledge before the feature distillation. Our DPK makes the performance of the student model positively correlated with that of the teacher model, which means that we can further boost the accuracy of students by applying larger teachers.
  arXiv  Detail & Related papers  (2022-06-13T11:52:13Z)
• Unified and Effective Ensemble Knowledge Distillation [92.67156911466397]
  Ensemble knowledge distillation can extract knowledge from multiple teacher models and encode it into a single student model. Many existing methods learn and distill the student model on labeled data only. We propose a unified and effective ensemble knowledge distillation method that distills a single student model from an ensemble of teacher models on both labeled and unlabeled data.
  arXiv  Detail & Related papers  (2022-04-01T16:15:39Z)
• Fixing the Teacher-Student Knowledge Discrepancy in Distillation [72.4354883997316]
  We propose a novel student-dependent distillation method, knowledge consistent distillation, which makes teacher's knowledge more consistent with the student. Our method is very flexible that can be easily combined with other state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-03-31T06:52:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.