Cross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification

• URL: http://arxiv.org/abs/2608.26771v1
• Date: Thu, 27 Aug 2026 08:02:05 GMT
• Title: Cross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification
• Abstract summary: Self-supervised vision foundation models such as DINOv2 provide strong features but are too large for field deployment.<n>We study cross-architecture knowledge distillation from a fine-tuned DINOv2 teacher to a compact bidirectional Visual State Space Model.<n>We identify and fix two training-stability problems that prevent the from-scratch SSM student from learning on limited data.
• Score: 10.122947222129108
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated tea leaf disease classification supports precision agriculture, yet deploying accurate models on edge devices remains challenging under tight compute budgets. Self-supervised vision foundation models such as DINOv2 provide strong features but are too large for field deployment, while lightweight models trained from scratch on small agricultural datasets often underfit. We study cross-architecture knowledge distillation (KD) from a fine-tuned DINOv2 teacher (Vision Transformer) to a compact bidirectional Visual State Space Model (LVSSM) student, an underexplored direction because the architectures use fundamentally different token-mixing mechanisms. We identify and fix two training-stability problems that prevent the from-scratch SSM student from learning on limited data: a single large patch-embedding convolution and a fusion layer that severs the residual path. With a progressive convolutional stem and gated bidirectional selective-scan block, the 4.45M-parameter student trains stably. Across three seeds, temperature-scaled logit distillation raises test accuracy from 92.32+/-2.14% to 95.41+/-1.17% (best single run: 96.20%; macro-F1: 94.45%), a +3.09 percentage-point mean gain. The student uses 5.0 times fewer parameters than the 22M-parameter teacher while retaining 98.3% of its accuracy. Ablations show that intermediate feature-alignment losses reduce accuracy, making simple logit-level KD the strongest configuration. A fair from-scratch comparison shows the gain is specific to students that start below the teacher. We report per-class metrics, confusion matrices, bootstrap confidence intervals, and FLOPs/latency measurements, and discuss limitations including the single-dataset scope and simplified non-official SSM implementation.

Related papers

• Towards Compact Unified Multimodal Tracking: Synergizing Knowledge Distillation with Structural Pruning [80.73561460934809]
  multimodal object tracking has achieved remarkable robustness by leveraging complementary sensor data.<n>We identify the prediction head as a critical but often overlooked efficiency bottleneck.<n>By strategically streamlining the decoder architecture, we unlock the potential for real-time inference.
  arXiv  Detail & Related papers  (2026-08-02T20:41:33Z)
• LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models [9.025382179940767]
  coarse-to-fine distillation framework with LInear FiTtingbased distillation (LIFT) and Piecewise Local Adaptive Coefficient Estimation (PLACE)<n>Experiments show LIFT and PLACE is effective across diffusion spaces (image/latent), backbones (U-Net/DiT), tasks (unconditional/conditional), datasets, and even extends to flow-based models such as MMDiT (SD3)
  arXiv  Detail & Related papers  (2026-05-19T12:03:53Z)
• Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions [11.509375725817195]
  We propose a two-stage knowledge distillation framework that mines high-value samples from existing data.<n>With augmented training on only 10.30% of filtered samples, we achieve MAP@3 of 0.9585 (+17.8%) on the MAP-Charting dataset.<n>Using only a 4B parameter model, we attain 84.38% accuracy on cross-topic tests of middle school algebra misconception benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T12:17:38Z)
• Distillation Traps and Guards: A Calibration Knob for LLM Distillability [54.90137955363471]
  We propose a post-hoc calibration method that enables control over a teacher's distillability via reinforcement fine-tuning.<n>Our objective combines task utility, KL anchor, and across-tokenizer calibration reward.<n> Experiments show that students distilled from distillable teachers outperform SFT and KD baselines.
  arXiv  Detail & Related papers  (2026-04-21T01:22:35Z)
• DKD-KAN: A Lightweight knowledge-distilled KAN intrusion detection framework, based on MLP and KAN [0.0]
  A light-weight intrusion detection framework is proposed that utilizes the Kolmogorov-Arnold Network (KAN) to capture complex features in the data.<n>A high-capacity KAN network is first trained to detect attacks performed on the test bed.<n>This model then serves as a teacher to guide a much smaller multilayer perceptron (MLP) student model via DKD.<n>The resulting DKD-MLP model contains only 2,522 and 1,622 parameters for WADI and SWaT datasets, which are significantly smaller than the number of parameters of the KAN teacher model.
  arXiv  Detail & Related papers  (2026-03-03T20:02:11Z)
• Post-Training Probability Manifold Correction via Structured SVD Pruning and Self-Referential Distillation [0.0]
  We introduce Sparse Knowledge Distillation (SparseKD), a method that compresses transformer models by combining structured SVD pruning with self-referential knowledge distillation.<n>SparseKD requires no external super-teacher, no architectural changes, and no custom inference kernels, making it immediately deployable with existing infrastructure.
  arXiv  Detail & Related papers  (2026-01-30T22:42:02Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• Unleashing the Power of One-Step Diffusion based Image Super-Resolution via a Large-Scale Diffusion Discriminator [81.81748032199813]
  Diffusion models have demonstrated excellent performance for real-world image super-resolution (Real-ISR)<n>We propose a new One-Step textbfDiffusion model with a larger-scale textbfDiscriminator for SR.<n>Our discriminator is able to distill noisy features from any time step of diffusion models in the latent space.
  arXiv  Detail & Related papers  (2024-10-05T16:41:36Z)
• One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation [69.65734716679925]
  Knowledge distillation has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. Most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family. We propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures.
  arXiv  Detail & Related papers  (2023-10-30T11:13:02Z)
• Two Independent Teachers are Better Role Model [7.001845833295753]
  We propose a new deep learning model called 3D-DenseUNet. It works as adaptable global aggregation blocks in down-sampling to solve the issue of spatial information loss. We also propose a new method called Two Independent Teachers, that summarizes the model weights instead of label predictions.
  arXiv  Detail & Related papers  (2023-06-09T08:22:41Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.