Zero-shot Cross-domain Knowledge Distillation: A Case study on YouTube Music

• URL: http://arxiv.org/abs/2603.28994v1
• Date: Mon, 30 Mar 2026 20:47:58 GMT
• Title: Zero-shot Cross-domain Knowledge Distillation: A Case study on YouTube Music
• Abstract summary: Knowledge Distillation (KD) has been widely used to improve the quality of latency sensitive models serving live traffic.<n>Cross-domain KD offers a cost-effective alternative by leveraging a teacher from a data-rich source domain.<n>We present a case study of using zero-shot cross-domain KD for multi-task ranking models.
• Score: 1.7419416877418235
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge Distillation (KD) has been widely used to improve the quality of latency sensitive models serving live traffic. However, applying KD in production recommender systems with low traffic is challenging: the limited amount of data restricts the teacher model size, and the cost of training a large dedicated teacher may not be justified. Cross-domain KD offers a cost-effective alternative by leveraging a teacher from a data-rich source domain, but introduces unique technical difficulties, as the features, user interfaces, and prediction tasks can significantly differ. We present a case study of using zero-shot cross-domain KD for multi-task ranking models, transferring knowledge from a (100x) large-scale video recommendation platform (YouTube) to a music recommendation application with significantly lower traffic. We share offline and live experiment results and present findings evaluating different KD techniques in this setting across two ranking models on the music app. Our results demonstrate that zero-shot cross-domain KD is a practical and effective approach to improve the performance of ranking models on low traffic surfaces.

Related papers

• Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails [16.73080036450313]
  Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments.<n> Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model.
  arXiv  Detail & Related papers  (2026-06-22T07:19:43Z)
• Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling [81.00825302340984]
  We introduce Speculative Knowledge Distillation (SKD) to generate high-quality training data on-the-fly.<n>In SKD, the student proposes tokens, and the teacher replaces poorly ranked ones based on its own distribution.<n>We evaluate SKD on various text generation tasks, including translation, summarization, math, and instruction following.
  arXiv  Detail & Related papers  (2024-10-15T06:51:25Z)
• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models [62.5501109475725]
  Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. This paper introduces Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
  arXiv  Detail & Related papers  (2024-09-19T07:05:26Z)
• Revisiting Knowledge Distillation for Autoregressive Language Models [88.80146574509195]
  We propose a simple yet effective adaptive teaching approach (ATKD) to improve the knowledge distillation (KD) The core of ATKD is to reduce rote learning and make teaching more diverse and flexible. Experiments on 8 LM tasks show that, with the help of ATKD, various baseline KD methods can achieve consistent and significant performance gains.
  arXiv  Detail & Related papers  (2024-02-19T07:01:10Z)
• Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning [3.1423836318272773]
  Knowledge distillation (KD) improves the performance of efficient and lightweight models.<n>Most existing KD techniques rely on Kullback-Leibler (KL) divergence.<n>We propose a Robustness-Reinforced Knowledge Distillation (R2KD) that leverages correlation distance and network pruning.
  arXiv  Detail & Related papers  (2023-11-23T11:34:48Z)
• Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation [10.899753512019933]
  Knowledge Distillation (KD) aims to optimize a lightweight network. KD mainly involves knowledge extraction and distillation strategies. This paper provides a comprehensive KD survey, including knowledge categories, distillation schemes and algorithms.
  arXiv  Detail & Related papers  (2023-06-19T03:42:44Z)
• Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization [29.113990037893597]
  Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) performance by transferring the knowledge from a larger model (a teacher) Existing KD techniques do not mitigate noise in the teacher's output: noisy behaviour distracts the student from learning more useful teacher. We propose a new KD method that addresses these problems compared to previous techniques.
  arXiv  Detail & Related papers  (2022-12-12T16:00:20Z)
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family [10.51711053229702]
  State-of-the-art results in deep learning have been improving steadily, in good part due to the use of larger models. While Knowledge Distillation (KD) theoretically enables small student models to emulate larger teacher models, in practice selecting a good student architecture requires considerable human expertise. In this paper, we propose to instead search for a family of student architectures sharing the property of being good at learning from a given teacher.
  arXiv  Detail & Related papers  (2021-11-05T15:20:37Z)
• How and When Adversarial Robustness Transfers in Knowledge Distillation? [137.11016173468457]
  This paper studies how and when the adversarial robustness can be transferred from a teacher model to a student model in Knowledge distillation (KD) We show that standard KD training fails to preserve adversarial robustness, and we propose KD with input gradient alignment (KDIGA) for remedy. Under certain assumptions, we prove that the student model using our proposed KDIGA can achieve at least the same certified robustness as the teacher model.
  arXiv  Detail & Related papers  (2021-10-22T21:30:53Z)
• Undistillable: Making A Nasty Teacher That CANNOT teach students [84.6111281091602]
  This paper introduces and investigates a concept called Nasty Teacher: a specially trained teacher network that yields nearly the same performance as a normal one. We propose a simple yet effective algorithm to build the nasty teacher, called self-undermining knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-16T08:41:30Z)
• Heterogeneous Knowledge Distillation using Information Flow Modeling [82.83891707250926]
  We propose a novel KD method that works by modeling the information flow through the various layers of the teacher model. The proposed method is capable of overcoming the aforementioned limitations by using an appropriate supervision scheme during the different phases of the training process.
  arXiv  Detail & Related papers  (2020-05-02T06:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.