Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation

• URL: http://arxiv.org/abs/2512.20908v1
• Date: Wed, 24 Dec 2025 03:19:05 GMT
• Title: Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation
• Authors: Kaiyuan Liu, Shaotian Yan, Rui Miao, Bing Wang, Chen Shen, Jun Zhang, Jieping Ye,
• Abstract summary: We introduce a cross-model Reasoning Distillation Provenance Tracing framework.<n>For each action produced by the distilled model, we obtain the predictive probabilities assigned by the teacher, the original student, and the distilled model under the same context.<n>We experimentally demonstrate that, in test-time contexts, the distilled model can indeed generate teacher-originated actions, which correlate with and plausibly explain observed performance on distilled model.
• Score: 44.23725347095524
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning distillation has attracted increasing attention. It typically leverages a large teacher model to generate reasoning paths, which are then used to fine-tune a student model so that it mimics the teacher's behavior in training contexts. However, previous approaches have lacked a detailed analysis of the origins of the distilled model's capabilities. It remains unclear whether the student can maintain consistent behaviors with the teacher in novel test-time contexts, or whether it regresses to its original output patterns, raising concerns about the generalization of distillation models. To analyse this question, we introduce a cross-model Reasoning Distillation Provenance Tracing framework. For each action (e.g., a sentence) produced by the distilled model, we obtain the predictive probabilities assigned by the teacher, the original student, and the distilled model under the same context. By comparing these probabilities, we classify each action into different categories. By systematically disentangling the provenance of each action, we experimentally demonstrate that, in test-time contexts, the distilled model can indeed generate teacher-originated actions, which correlate with and plausibly explain observed performance on distilled model. Building on this analysis, we further propose a teacher-guided data selection method. Unlike prior approach that rely on heuristics, our method directly compares teacher-student divergences on the training data, providing a principled selection criterion. We validate the effectiveness of our approach across multiple representative teacher models and diverse student models. The results highlight the utility of our provenance-tracing framework and underscore its promise for reasoning distillation. We hope to share Reasoning Distillation Provenance Tracing and our insights into reasoning distillation with the community.

Related papers

• Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation [84.38105530043741]
  We propose Warmup-Distill, which aligns the distillation of the student to that of the teacher in advance of distillation.<n>Experiments on the seven benchmarks demonstrate that Warmup-Distill could provide a warmup student more suitable for distillation.
  arXiv  Detail & Related papers  (2025-02-17T12:58:12Z)
• Towards Training One-Step Diffusion Models Without Distillation [72.80423908458772]
  We introduce a family of new training methods that entirely forgo teacher score supervision.<n>We find that initializing the student model with the teacher's weights remains critical.
  arXiv  Detail & Related papers  (2025-02-11T23:02:14Z)
• Progressive distillation induces an implicit curriculum [44.528775476168654]
  A better teacher does not always yield a better student, to which a common mitigation is to use additional supervision from several teachers. One empirically validated variant of this principle is progressive distillation, where the student learns from successive intermediate checkpoints of the teacher. Using sparse parity as a sandbox, we identify an implicit curriculum as one mechanism through which progressive distillation accelerates the student's learning.
  arXiv  Detail & Related papers  (2024-10-07T19:49:24Z)
• Knowledge Distillation with Refined Logits [31.205248790623703]
  We introduce Refined Logit Distillation (RLD) to address the limitations of current logit distillation methods.<n>Our approach is motivated by the observation that even high-performing teacher models can make incorrect predictions.<n>Our method can effectively eliminate misleading information from the teacher while preserving crucial class correlations.
  arXiv  Detail & Related papers  (2024-08-14T17:59:32Z)
• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)
• A Generic Approach for Reproducible Model Distillation [2.457924087844968]
  We develop a generic approach for stable model distillation based on central limit theorem for the average loss. We demonstrate the application of our proposed approach on three commonly used intelligible models: decision trees, falling rule lists and symbolic regression.
  arXiv  Detail & Related papers  (2022-11-22T23:38:02Z)
• Teacher's pet: understanding and mitigating biases in distillation [61.44867470297283]
  Several works have shown that distillation significantly boosts the student's overall performance. However, are these gains uniform across all data subgroups? We show that distillation can harm performance on certain subgroups. We present techniques which soften the teacher influence for subgroups where it is less reliable.
  arXiv  Detail & Related papers  (2021-06-19T13:06:25Z)
• Joint learning of interpretation and distillation [7.412850488684037]
  This paper conducts an empirical study on the new approach to explaining each prediction of GBDT2NN. Experiments on several benchmarks show that the proposed methods achieve better performance on both explanations and predictions.
  arXiv  Detail & Related papers  (2020-05-24T02:01:22Z)
• Why distillation helps: a statistical perspective [69.90148901064747]
  Knowledge distillation is a technique for improving the performance of a simple "student" model. While this simple approach has proven widely effective, a basic question remains unresolved: why does distillation help? We show how distillation complements existing negative mining techniques for extreme multiclass retrieval.
  arXiv  Detail & Related papers  (2020-05-21T01:49:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.