Memorization Dynamics in Knowledge Distillation for Language Models

• URL: http://arxiv.org/abs/2601.15394v1
• Date: Wed, 21 Jan 2026 19:04:40 GMT
• Title: Memorization Dynamics in Knowledge Distillation for Language Models
• Abstract summary: We study memorization across the KD pipeline using three large language model (LLM) families and three datasets.<n>We find: (1) distilled models significantly less training data than standard fine-tuning (reducing by more than 50%); (2) some examples are inherently easier to memorize and account for a large fraction of memorization during distillation (over 95%); and (3) student memorization is predictable prior to distillation using features based on zlib entropy, KL divergence, and perplexity.
• Score: 16.76814083270349
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge Distillation (KD) is increasingly adopted to transfer capabilities from large language models to smaller ones, offering significant improvements in efficiency and utility while often surpassing standard fine-tuning. Beyond performance, KD is also explored as a privacy-preserving mechanism to mitigate the risk of training data leakage. While training data memorization has been extensively studied in standard pre-training and fine-tuning settings, its dynamics in a knowledge distillation setup remain poorly understood. In this work, we study memorization across the KD pipeline using three large language model (LLM) families (Pythia, OLMo-2, Qwen-3) and three datasets (FineWeb, Wikitext, Nemotron-CC-v2). We find: (1) distilled models memorize significantly less training data than standard fine-tuning (reducing memorization by more than 50%); (2) some examples are inherently easier to memorize and account for a large fraction of memorization during distillation (over ~95%); (3) student memorization is predictable prior to distillation using features based on zlib entropy, KL divergence, and perplexity; and (4) while soft and hard distillation have similar overall memorization rates, hard distillation poses a greater risk: it inherits $2.7\times$ more teacher-specific examples than soft distillation. Overall, we demonstrate that distillation can provide both improved generalization and reduced memorization risks compared to standard fine-tuning.

Related papers

• Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition [10.351322944851095]
  We introduce Label-Guided Knowledge Distillation (LGKD) to guide the distillation of student model features using ground truth labels.<n>Our method entails two components: sample-wise distillation and class-wise distillation, enabling the student model to learn feature representation of the teacher model at two levels.
  arXiv  Detail & Related papers  (2026-09-11T16:21:21Z)
• EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer [72.31900638260758]
  Large language models (LLMs) have achieved remarkable performance across diverse domains.<n> Knowledge distillation offers a promising solution by transferring knowledge from a large teacher model to a smaller student model.<n>We propose an entropy-based adaptive distillation strategy that dynamically adjusts the training process at the token level.
  arXiv  Detail & Related papers  (2026-05-03T06:05:51Z)
• On the Memorization of Consistency Distillation for Diffusion Models [31.08329979854717]
  We analyze how consistency distillation reshapes memorization behavior in diffusion models.<n>We show that consistency distillation significantly reduces transferred memorization in the student while preserving, and sometimes improving, sample quality.<n>Our findings suggest that distillation can serve not only as an acceleration tool, but also as a mechanism for improving the memorization-generalization trade-off.
  arXiv  Detail & Related papers  (2026-04-26T06:22:53Z)
• Distillation Traps and Guards: A Calibration Knob for LLM Distillability [54.90137955363471]
  We propose a post-hoc calibration method that enables control over a teacher's distillability via reinforcement fine-tuning.<n>Our objective combines task utility, KL anchor, and across-tokenizer calibration reward.<n> Experiments show that students distilled from distillable teachers outperform SFT and KD baselines.
  arXiv  Detail & Related papers  (2026-04-21T01:22:35Z)
• From Teacher to Student: Tracking Memorization Through Model Distillation [0.9065034043031668]
  Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security.<n>In this study, we explore how different knowledge distillation (KD) methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.
  arXiv  Detail & Related papers  (2025-06-19T09:44:25Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Efficient Knowledge Injection in LLMs via Self-Distillation [50.24554628642021]
  This paper proposes utilizing prompt distillation to internalize new factual knowledge from free-form documents.<n>We show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG.
  arXiv  Detail & Related papers  (2024-12-19T15:44:01Z)
• Small Scale Data-Free Knowledge Distillation [37.708282211941416]
  We propose Small Scale Data-free Knowledge Distillation SSD-KD. SSD-KD balances synthetic samples and a priority sampling function to select proper samples. It can perform distillation training conditioned on an extremely small scale of synthetic samples.
  arXiv  Detail & Related papers  (2024-06-12T05:09:41Z)
• Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss [32.816725317261934]
  This work studies knowledge distillation (KD) and addresses its constraints for recurrent neural network transducer (RNN-T) models. We show that a sequence-level KD, full-sum distillation, outperforms other distillation methods for RNN-T models. We also propose a variant of full-sum distillation that distills the sequence discriminative knowledge of the teacher leading to further improvement in WER.
  arXiv  Detail & Related papers  (2023-03-10T14:46:23Z)
• Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation [72.70058049274664]
  We present Referee, a novel framework for sentence summarization that can be trained reference-free (i.e., requiring no gold summaries for supervision) Our work is the first to demonstrate that reference-free, controlled sentence summarization is feasible via the conceptual framework of Symbolic Knowledge Distillation.
  arXiv  Detail & Related papers  (2022-10-25T07:07:54Z)
• Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation [74.67594286008317]
  This article addresses the problem of distilling knowledge from a large teacher model to a slim student network for LiDAR semantic segmentation. We propose the Point-to-Voxel Knowledge Distillation (PVD), which transfers the hidden knowledge from both point level and voxel level.
  arXiv  Detail & Related papers  (2022-06-05T05:28:32Z)
• Quantifying Memorization Across Neural Language Models [61.58529162310382]
  Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others). We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data.
  arXiv  Detail & Related papers  (2022-02-15T18:48:31Z)
• Pre-trained Summarization Distillation [121.14806854092672]
  Recent work on distilling BERT for classification and regression tasks shows strong performance using direct knowledge distillation. Alternatively, machine translation practitioners distill using pseudo-labeling, where a small model is trained on the translations of a larger model. A third, simpler approach is to'shrink and fine-tune' (SFT), which avoids any explicit distillation by copying parameters to a smaller student model and then fine-tuning.
  arXiv  Detail & Related papers  (2020-10-24T23:15:43Z)
• Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings [5.929956715430167]
  Transfer learning with large-scale language models (LM) has led to dramatic performance improvements across a broad range of natural language understanding tasks. The size and memory footprint of these large LMs makes them difficult to deploy in many scenarios. Recent research points to knowledge distillation as a potential solution, showing that when training data for a given task is abundant, it is possible to distill a large (teacher) LM into a small task-specific (student) network with minimal loss of performance.
  arXiv  Detail & Related papers  (2020-01-25T08:20:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.