Continual Collaborative Distillation for Recommender System

• URL: http://arxiv.org/abs/2405.19046v4
• Date: Wed, 26 Jun 2024 00:21:43 GMT
• Title: Continual Collaborative Distillation for Recommender System
• Authors: Gyuseok Lee, SeongKu Kang, Wonbin Kweon, Hwanjo Yu,
• Abstract summary: We propose a systematic approach to operating the teacher-student KD in a non-stationary data stream. We propose Continual Collaborative Distillation (CCD) framework, where both the teacher and the student continually and collaboratively evolve along the data stream. We validate the effectiveness of CCD through extensive quantitative, ablative, and exploratory experiments on two real-world datasets.
• Score: 20.10443406772368
• License:
• Abstract: Knowledge distillation (KD) has emerged as a promising technique for addressing the computational challenges associated with deploying large-scale recommender systems. KD transfers the knowledge of a massive teacher system to a compact student model, to reduce the huge computational burdens for inference while retaining high accuracy. The existing KD studies primarily focus on one-time distillation in static environments, leaving a substantial gap in their applicability to real-world scenarios dealing with continuously incoming users, items, and their interactions. In this work, we delve into a systematic approach to operating the teacher-student KD in a non-stationary data stream. Our goal is to enable efficient deployment through a compact student, which preserves the high performance of the massive teacher, while effectively adapting to continuously incoming data. We propose Continual Collaborative Distillation (CCD) framework, where both the teacher and the student continually and collaboratively evolve along the data stream. CCD facilitates the student in effectively adapting to new data, while also enabling the teacher to fully leverage accumulated knowledge. We validate the effectiveness of CCD through extensive quantitative, ablative, and exploratory experiments on two real-world datasets. We expect this research direction to contribute to narrowing the gap between existing KD studies and practical applications, thereby enhancing the applicability of KD in real-world systems.

Related papers

• Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling [81.00825302340984]
  We introduce Speculative Knowledge Distillation (SKD) to generate high-quality training data on-the-fly. In SKD, the student proposes tokens, and the teacher replaces poorly ranked ones based on its own distribution. We evaluate SKD on various text generation tasks, including translation, summarization, math, and instruction following.
  arXiv  Detail & Related papers  (2024-10-15T06:51:25Z)
• Applications of Knowledge Distillation in Remote Sensing: A Survey [3.481234252899159]
  Knowledge distillation (KD) is a technique developed to transfer knowledge from a complex, often cumbersome model (teacher) to a more compact and efficient model (student) The article provides a comprehensive taxonomy of KD techniques, where each category is critically analyzed to demonstrate the breadth and depth of the alternative options. The review discusses the challenges and limitations of KD in RS, including practical constraints and prospective future directions.
  arXiv  Detail & Related papers  (2024-09-18T16:30:49Z)
• Bridging the Gap: Unpacking the Hidden Challenges in Knowledge Distillation for Online Ranking Systems [13.437632008276552]
  Knowledge Distillation (KD) is a powerful approach for compressing a large model into a smaller, more efficient model. We present a robust KD system developed and rigorously evaluated on multiple large-scale personalized video recommendation systems within Google.
  arXiv  Detail & Related papers  (2024-08-26T23:01:48Z)
• Relative Difficulty Distillation for Semantic Segmentation [54.76143187709987]
  We propose a pixel-level KD paradigm for semantic segmentation named Relative Difficulty Distillation (RDD) RDD allows the teacher network to provide effective guidance on learning focus without additional optimization goals. Our research showcases that RDD can integrate with existing KD methods to improve their upper performance bound.
  arXiv  Detail & Related papers  (2024-07-04T08:08:25Z)
• Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning [3.1423836318272773]
  Knowledge distillation (KD) improves the performance of efficient and lightweight models. Most existing KD techniques rely on Kullback-Leibler (KL) divergence. We propose a Robustness-Reinforced Knowledge Distillation (R2KD) that leverages correlation distance and network pruning.
  arXiv  Detail & Related papers  (2023-11-23T11:34:48Z)
• Comparative Knowledge Distillation [102.35425896967791]
  Traditional Knowledge Distillation (KD) assumes readily available access to teacher models for frequent inference. We propose Comparative Knowledge Distillation (CKD), which encourages student models to understand the nuanced differences in a teacher model's interpretations of samples. CKD consistently outperforms state of the art data augmentation and KD techniques.
  arXiv  Detail & Related papers  (2023-11-03T21:55:33Z)
• Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation [65.62538699160085]
  We propose a Directed Acyclic Graph Factorization Machine (KD-DAGFM) to learn the high-order feature interactions from existing complex interaction models for CTR prediction via Knowledge Distillation. KD-DAGFM achieves the best performance with less than 21.5% FLOPs of the state-of-the-art method on both online and offline experiments.
  arXiv  Detail & Related papers  (2022-11-21T03:09:42Z)
• Semi-Online Knowledge Distillation [2.373824287636486]
  Conventional knowledge distillation (KD) is to transfer knowledge from a large and well pre-trained teacher network to a small student network. Deep mutual learning (DML) has been proposed to help student networks learn collaboratively and simultaneously. We propose a Semi-Online Knowledge Distillation (SOKD) method that effectively improves the performance of the student and the teacher.
  arXiv  Detail & Related papers  (2021-11-23T09:44:58Z)
• How and When Adversarial Robustness Transfers in Knowledge Distillation? [137.11016173468457]
  This paper studies how and when the adversarial robustness can be transferred from a teacher model to a student model in Knowledge distillation (KD) We show that standard KD training fails to preserve adversarial robustness, and we propose KD with input gradient alignment (KDIGA) for remedy. Under certain assumptions, we prove that the student model using our proposed KDIGA can achieve at least the same certified robustness as the teacher model.
  arXiv  Detail & Related papers  (2021-10-22T21:30:53Z)
• Knowledge Distillation Thrives on Data Augmentation [65.58705111863814]
  Knowledge distillation (KD) is a general deep neural network training framework that uses a teacher model to guide a student model. Many works have explored the rationale for its success, however, its interplay with data augmentation (DA) has not been well recognized so far. In this paper, we are motivated by an interesting observation in classification: KD loss can benefit from extended training iterations while the cross-entropy loss does not. We show this disparity arises because of data augmentation: KD loss can tap into the extra information from different input views brought by DA.
  arXiv  Detail & Related papers  (2020-12-05T00:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.