Reference-Based Distillation Detection in LLMs

• URL: http://arxiv.org/abs/2607.09692v1
• Date: Fri, 19 Jun 2026 06:59:19 GMT
• Title: Reference-Based Distillation Detection in LLMs
• Abstract summary: We introduce a distillation detection method based on reference-based membership inference.<n>We develop a hybrid evaluation spanning both controlled distillation experiments and real-world models.<n>Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.
• Score: 17.877444310396758
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations. This motivates a fundamental question: can we detect whether a model was distilled from another? We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in a reference-based setting: given a model and an earlier-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint. We introduce a distillation detection method based on reference-based membership inference. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs. We additionally identify a distinctive glyph-level signal specific to o1/o3 models. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real-world models. Across both settings, our approach recovers the true teacher with near-perfect accuracy in single-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open-world settings where no teacher is guaranteed to be present among the candidates. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.

Related papers

• Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation [44.23725347095524]
  We introduce a cross-model Reasoning Distillation Provenance Tracing framework.<n>For each action produced by the distilled model, we obtain the predictive probabilities assigned by the teacher, the original student, and the distilled model under the same context.<n>We experimentally demonstrate that, in test-time contexts, the distilled model can indeed generate teacher-originated actions, which correlate with and plausibly explain observed performance on distilled model.
  arXiv  Detail & Related papers  (2025-12-24T03:19:05Z)
• Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation [84.38105530043741]
  We propose Warmup-Distill, which aligns the distillation of the student to that of the teacher in advance of distillation.<n>Experiments on the seven benchmarks demonstrate that Warmup-Distill could provide a warmup student more suitable for distillation.
  arXiv  Detail & Related papers  (2025-02-17T12:58:12Z)
• Towards Training One-Step Diffusion Models Without Distillation [72.80423908458772]
  We introduce a family of new training methods that entirely forgo teacher score supervision.<n>We find that initializing the student model with the teacher's weights remains critical.
  arXiv  Detail & Related papers  (2025-02-11T23:02:14Z)
• Training on the Test Model: Contamination in Ranking Distillation [14.753216172912968]
  We investigate the effect of a contaminated teacher model in a distillation setting. We find that contamination occurs even when the test data represents a small fraction of the teacher's training samples.
  arXiv  Detail & Related papers  (2024-11-04T17:11:14Z)
• Knowledge Distillation with Refined Logits [31.205248790623703]
  We introduce Refined Logit Distillation (RLD) to address the limitations of current logit distillation methods.<n>Our approach is motivated by the observation that even high-performing teacher models can make incorrect predictions.<n>Our method can effectively eliminate misleading information from the teacher while preserving crucial class correlations.
  arXiv  Detail & Related papers  (2024-08-14T17:59:32Z)
• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)
• Unbiased Knowledge Distillation for Recommendation [66.82575287129728]
  Knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge to supervise the learning of a compact student model. We find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation.
  arXiv  Detail & Related papers  (2022-11-27T05:14:03Z)
• A Generic Approach for Reproducible Model Distillation [2.457924087844968]
  We develop a generic approach for stable model distillation based on central limit theorem for the average loss. We demonstrate the application of our proposed approach on three commonly used intelligible models: decision trees, falling rule lists and symbolic regression.
  arXiv  Detail & Related papers  (2022-11-22T23:38:02Z)
• Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation [72.70058049274664]
  We present Referee, a novel framework for sentence summarization that can be trained reference-free (i.e., requiring no gold summaries for supervision) Our work is the first to demonstrate that reference-free, controlled sentence summarization is feasible via the conceptual framework of Symbolic Knowledge Distillation.
  arXiv  Detail & Related papers  (2022-10-25T07:07:54Z)
• Why distillation helps: a statistical perspective [69.90148901064747]
  Knowledge distillation is a technique for improving the performance of a simple "student" model. While this simple approach has proven widely effective, a basic question remains unresolved: why does distillation help? We show how distillation complements existing negative mining techniques for extreme multiclass retrieval.
  arXiv  Detail & Related papers  (2020-05-21T01:49:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.