DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer

• URL: http://arxiv.org/abs/2603.15166v1
• Date: Mon, 16 Mar 2026 12:00:31 GMT
• Title: DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer
• Abstract summary: Large-scale Vision-Language Models (VLMs) encode rich multimodal semantics that are highly beneficial for fine-grained visual categorization.<n>Their prohibitive computational cost hinders practical deployment in resource-constrained environments.<n>We propose Distillation with Adaptive Intermediate Teacher transfer (DAIT) to facilitate adaptive knowledge transfer from VLMs to lightweight students.
• Score: 10.030998208145807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale Vision-Language Models (VLMs) encode rich multimodal semantics that are highly beneficial for fine-grained visual categorization (FGVC). However, their prohibitive computational cost hinders practical deployment in resource-constrained environments. Although knowledge distillation contributes to transferring VLMs capacity to lightweight classifiers, conventional distillation mechanisms, which directly transfer from a generic VLM to a compact student, often yield suboptimal results due to severe architectural misalignment and introducing task-irrelevant information. To alleviate this limitation, we propose Distillation with Adaptive Intermediate Teacher transfer (DAIT) in this study, facilitating adaptive knowledge transfer from VLMs to lightweight students. DAIT introduces a trainable intermediate teacher that learns to transfer frozen VLMs representations under explicit supervision from the target fine-grained task. This intermediate teacher adaptively enhances discriminative visual cues, thereby producing compact and task-aligned knowledge that can be reliably distilled into lightweight models. Extensive evaluations on multiple FGVC benchmarks with diverse student architectures demonstrate that our method achieves respective performance gains of 12.63% and 8.34% on FGVC-Aircraft and CUB-200-2011 datasets, establishing DAIT as a principled paradigm for transferring from general-purpose VLMS to deployable fine-grained recognition models.

Related papers

• SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation [1.1617094867403568]
  Large-scale vision foundation models have driven substantial gains on dense prediction tasks such as semantic segmentation.<n>Modern foundation teachers are predominantly transformer-based that encode global context, whereas efficient students are typically convolutional networks with locally biased receptive fields.<n>Existing distillation methods largely assume architectural homogeneity and rely on direct feature mimicry, which fails to bridge this representational gap.<n>We propose SWARD, a knowledge distillation framework that addresses this gap through two complementary mechanisms.
  arXiv  Detail & Related papers  (2026-05-31T04:29:31Z)
• AgriKD: Cross-Architecture Knowledge Distillation for Efficient Leaf Disease Classification [0.05599792629509228]
  AgriKD is a cross-architecture knowledge distillation framework for efficient edge deployment.<n>It transfers knowledge from a Vision Transformer (ViT) teacher to a compact convolutional student model.<n>It achieves consistent predictive performance with negligible accuracy.
  arXiv  Detail & Related papers  (2026-05-02T09:58:57Z)
• Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Models [16.045233710264842]
  Knowledge Vector Weakening (KVW) is a training-free unlearning method that directly intervenes in the full model without gradient computation.<n> Experiments on the MLLMU and CLEAR benchmarks demonstrate that KVW achieves a stable forget-retain trade-off while significantly improving computational efficiency.
  arXiv  Detail & Related papers  (2026-01-29T14:41:01Z)
• VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation [67.98620973023709]
  VOLD is a framework to transfer reasoning capabilities from text-only teacher models to VLM student models.<n>We show that VOLD outperforms the baseline model significantly and improves over the state of the art by a margin.
  arXiv  Detail & Related papers  (2025-10-27T16:32:12Z)
• LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions [3.5268162595059076]
  This paper proposes a Multi-dimensional Dynamic Prompt Routing (MDPR) framework for vision-language models (VLMs)<n>MDPR constructs a comprehensive knowledge base for classes, spanning five visual-semantic dimensions.<n>Experiments on long-tailed benchmarks, including CIFAR-LT, ImageNet-LT, and Places-LT, demonstrate that MDPR achieves comparable results with current SOTA methods.
  arXiv  Detail & Related papers  (2025-08-21T16:12:06Z)
• Seeing Further on the Shoulders of Giants: Knowledge Inheritance for Vision Foundation Models [54.517276878748305]
  Vision foundation models (VFMs) are predominantly developed using data-centric methods.<n>Many open-source vision models have been pretrained on domain-specific data.<n>We present a new model-driven approach for training VFMs through joint knowledge transfer and preservation.
  arXiv  Detail & Related papers  (2025-08-20T13:30:23Z)
• DART: Dual Adaptive Refinement Transfer for Open-Vocabulary Multi-Label Recognition [59.203152078315235]
  Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image.<n> Vision-Language Pre-training models offer a strong open-vocabulary foundation, but struggle with fine-grained localization under weak supervision.<n>We propose the Dual Adaptive Refinement Transfer (DART) framework to overcome these limitations.
  arXiv  Detail & Related papers  (2025-08-07T17:22:33Z)
• VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models [84.84277196012907]
  We propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes.<n>We validate VLsI across ten challenging vision-language benchmarks, achieving notable performance gains (11.0% for 2B and 17.4% for 7B) over GPT-4V.
  arXiv  Detail & Related papers  (2024-12-02T18:58:25Z)
• DDK: Distilling Domain Knowledge for Efficient Large Language Models [40.839056203329136]
  Knowledge Distillation (KD) has emerged as an effective strategy to improve the performance of a smaller language model. This paper introduces DDK, which adjusts the composition of the distillation dataset according to the domain performance differences between the teacher and student models. Extensive evaluations show that DDK significantly improves the performance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by a large margin.
  arXiv  Detail & Related papers  (2024-07-23T03:47:28Z)
• VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness [56.87603097348203]
  VeCAF uses labels and natural language annotations to perform parametric data selection for PVM finetuning. VeCAF incorporates the finetuning objective to select significant data points that effectively guide the PVM towards faster convergence. On ImageNet, VeCAF uses up to 3.3x less training batches to reach the target performance compared to full finetuning.
  arXiv  Detail & Related papers  (2024-01-15T17:28:37Z)
• Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification [35.277880733198586]
  Vision-Language Models (VLMs) are trained on large amounts of image-text pairs, resulting in remarkable generalization across several data distributions. We propose Vision-Language to Vision - Align, Distill, Predict (VL2V-ADiP), which first aligns the vision and language modalities of the teacher model with the vision modality of a pre-trained student model. This maximally retains the pre-trained features of the student, while also incorporating the rich representations of the VLM image encoder and the superior generalization of the text embeddings.
  arXiv  Detail & Related papers  (2023-10-12T11:59:54Z)
• LIB-KD: Teaching Inductive Bias for Efficient Vision Transformer Distillation and Compression [4.0120180943504655]
  Vision Transformers (ViTs) offer the tantalising prospect of unified information processing across visual and textual domains.<n>We introduce an innovative ensemble-based distillation approach that distils inductive bias from complementary lightweight teacher models to make their applications practical.
  arXiv  Detail & Related papers  (2023-09-30T13:21:29Z)
• Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation [56.053397775016755]
  We propose a sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teacher detectors to a given lightweight student. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students.
  arXiv  Detail & Related papers  (2023-08-17T17:17:08Z)
• Unifying Synergies between Self-supervised Learning and Dynamic Computation [53.66628188936682]
  We present a novel perspective on the interplay between SSL and DC paradigms. We show that it is feasible to simultaneously learn a dense and gated sub-network from scratch in a SSL setting. The co-evolution during pre-training of both dense and gated encoder offers a good accuracy-efficiency trade-off.
  arXiv  Detail & Related papers  (2023-01-22T17:12:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.