Knowledge Distillation Detection for Open-weights Models

• URL: http://arxiv.org/abs/2510.02302v1
• Date: Thu, 02 Oct 2025 17:59:14 GMT
• Title: Knowledge Distillation Detection for Open-weights Models
• Authors: Qin Shi, Amber Yijia Zheng, Qifan Song, Raymond A. Yeh,
• Abstract summary: We introduce a model-agnostic framework that combines data-free input synthesis and statistical score for detecting distillation.<n>Our approach is applicable to both classification and generative models.
• Score: 30.640429431816614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose the task of knowledge distillation detection, which aims to determine whether a student model has been distilled from a given teacher, under a practical setting where only the student's weights and the teacher's API are available. This problem is motivated by growing concerns about model provenance and unauthorized replication through distillation. To address this task, we introduce a model-agnostic framework that combines data-free input synthesis and statistical score computation for detecting distillation. Our approach is applicable to both classification and generative models. Experiments on diverse architectures for image classification and text-to-image generation show that our method improves detection accuracy over the strongest baselines by 59.6% on CIFAR-10, 71.2% on ImageNet, and 20.0% for text-to-image generation. The code is available at https://github.com/shqii1j/distillation_detection.

Related papers

• Efficient Face Image Quality Assessment via Self-training and Knowledge Distillation [51.43664253596246]
  Face image quality assessment (FIQA) is essential for various face-related applications.<n>We aim to develop a computationally efficient FIQA method that can be easily deployed in real-world applications.
  arXiv  Detail & Related papers  (2025-07-21T15:17:01Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Improved Distribution Matching Distillation for Fast Image Synthesis [54.72356560597428]
  We introduce DMD2, a set of techniques that lift this limitation and improve DMD training. First, we eliminate the regression loss and the need for expensive dataset construction. Second, we integrate a GAN loss into the distillation procedure, discriminating between generated samples and real images.
  arXiv  Detail & Related papers  (2024-05-23T17:59:49Z)
• Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation [56.053397775016755]
  We propose a sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teacher detectors to a given lightweight student. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students.
  arXiv  Detail & Related papers  (2023-08-17T17:17:08Z)
• EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval [83.79667141681418]
  Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR) We propose a novel distillation approach that leverages the relative geometry among queries and documents learned by the large teacher model. We show that our approach successfully distills from both dual-encoder (DE) and cross-encoder (CE) teacher models to 1/10th size asymmetric students that can retain 95-97% of the teacher performance.
  arXiv  Detail & Related papers  (2023-01-27T22:04:37Z)
• Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation [66.25738680429463]
  Knowledge Distillation (KD) for object detection aims to train a compact detector by transferring knowledge from a teacher model. We propose inconsistent knowledge distillation (IKD) which aims to distill knowledge inherent in the teacher model's counter-intuitive perceptions. Our method outperforms state-of-the-art KD baselines on one-stage, two-stage and anchor-free object detectors.
  arXiv  Detail & Related papers  (2022-09-20T16:36:28Z)
• Context Unaware Knowledge Distillation for Image Retrieval [11.38957822323395]
  Existing knowledge distillation methods use logits and other features of the deep (teacher) model. We propose context unaware knowledge distillation that uses the knowledge of the teacher model without fine-tuning it on the target context.
  arXiv  Detail & Related papers  (2022-07-19T04:51:39Z)
• RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models [11.906350752875342]
  Student models acquired by knowledge distillation suffer from adversarial attacks, which limits their usage in security sensitive scenarios. A directed acyclic graph based search space is built and an evolutionary search strategy is utilized to guide the searching approach. Experimental results show that RoSearch can improve robustness of student models from 7%18% up to 45.8%47.8% on different datasets.
  arXiv  Detail & Related papers  (2021-06-07T13:38:16Z)
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones [40.33419553042038]
  We propose to improve existing baseline networks via knowledge distillation from off-the-shelf pre-trained big powerful models. Our solution performs distillation by only driving prediction of the student model consistent with that of the teacher model. We empirically find that such simple distillation settings perform extremely effective, for example, the top-1 accuracy on ImageNet-1k validation set of MobileNetV3-large and ResNet50-D can be significantly improved.
  arXiv  Detail & Related papers  (2021-03-10T09:32:44Z)
• General Instance Distillation for Object Detection [12.720908566642812]
  RetinaNet with ResNet-50 achieves 39.1% in mAP with GID on dataset, which surpasses the baseline 36.2% by 2.9%, and even better than the ResNet-101 based teacher model with 38.1% AP.
  arXiv  Detail & Related papers  (2021-03-03T11:41:26Z)
• Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation [19.778192371420793]
  A data-free adversarial distillation framework deploys a generative network to transfer the teacher model's knowledge to the student model. We add an activation regularizer and a virtual adversarial method to improve the data generation efficiency. Our model's accuracy is 13.8% higher than the state-of-the-art data-free method on CIFAR-100.
  arXiv  Detail & Related papers  (2021-02-23T11:37:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.