Knowledge Injection via Prompt Distillation

• URL: http://arxiv.org/abs/2412.14964v1
• Date: Thu, 19 Dec 2024 15:44:01 GMT
• Title: Knowledge Injection via Prompt Distillation
• Authors: Kalle Kujanpää, Harri Valpola, Alexander Ilin,
• Abstract summary: We propose a new fine-tuning technique for learning new knowledge and show that it can reach the performance of RAG. The proposed method is based on the self-distillation approach, which we call prompt distillation.
• Score: 48.66694777454059
• License:
• Abstract: In many practical applications, large language models (LLMs) need to incorporate new knowledge not present in their pre-training data. The primary methods for this are fine-tuning and retrieval-augmented generation (RAG). Although RAG has emerged as the industry standard for knowledge injection, fine-tuning has not yet achieved comparable success. In this paper, we propose a new fine-tuning technique for learning new knowledge and show that it can reach the performance of RAG. The proposed method is based on the self-distillation approach, which we call prompt distillation. First, we generate question-answer pairs about the new knowledge. Then, we fine-tune a student model on the question-answer pairs to imitate the output distributions of a teacher model, which additionally receives the new knowledge in its prompt. The student model is identical to the teacher, except it is equipped with a LoRA adapter. This training procedure facilitates distilling the new knowledge from the teacher's prompt into the student's weights.

Related papers

• AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression [26.474962405945316]
  We present a novel attribution-driven knowledge distillation approach to compress pre-trained language models. To enhance the knowledge transfer of model reasoning and generalization, we explore multi-view attribution distillation on all potential decisions of the teacher.
  arXiv  Detail & Related papers  (2023-05-17T07:40:12Z)
• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)
• Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection [35.515135913846386]
  We propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. Experimental results on the GLUE datasets show that our method outperforms several strong knowledge distillation baselines significantly.
  arXiv  Detail & Related papers  (2023-02-01T13:40:19Z)
• Unbiased Knowledge Distillation for Recommendation [66.82575287129728]
  Knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge to supervise the learning of a compact student model. We find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation.
  arXiv  Detail & Related papers  (2022-11-27T05:14:03Z)
• Multi-Teacher Knowledge Distillation for Incremental Implicitly-Refined Classification [37.14755431285735]
  We propose a novel Multi-Teacher Knowledge Distillation (MTKD) strategy for incremental learning. To preserve the superclass knowledge, we use the initial model as a superclass teacher to distill the superclass knowledge for the student model. We propose a post-processing mechanism, called as Top-k prediction restriction to reduce the redundant predictions.
  arXiv  Detail & Related papers  (2022-02-23T09:51:40Z)
• Dynamic Rectification Knowledge Distillation [0.0]
  Dynamic Rectification Knowledge Distillation (DR-KD) is a knowledge distillation framework. DR-KD transforms the student into its own teacher, and if the self-teacher makes wrong predictions while distilling information, the error is rectified prior to the knowledge being distilled. Our proposed DR-KD performs remarkably well in the absence of a sophisticated cumbersome teacher model.
  arXiv  Detail & Related papers  (2022-01-27T04:38:01Z)
• Learning Student-Friendly Teacher Networks for Knowledge Distillation [50.11640959363315]
  We propose a novel knowledge distillation approach to facilitate the transfer of dark knowledge from a teacher to a student. Contrary to most of the existing methods that rely on effective training of student models given pretrained teachers, we aim to learn the teacher models that are friendly to students.
  arXiv  Detail & Related papers  (2021-02-12T07:00:17Z)
• Reinforced Multi-Teacher Selection for Knowledge Distillation [54.72886763796232]
  knowledge distillation is a popular method for model compression. Current methods assign a fixed weight to a teacher model in the whole distillation. Most of the existing methods allocate an equal weight to every teacher model. In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled.
  arXiv  Detail & Related papers  (2020-12-11T08:56:39Z)
• DE-RRD: A Knowledge Distillation Framework for Recommender System [16.62204445256007]
  We propose a knowledge distillation framework for recommender system, called DE-RRD. It enables the student model to learn from the latent knowledge encoded in the teacher model as well as from the teacher's predictions. Our experiments show that DE-RRD outperforms the state-of-the-art competitors and achieves comparable or even better performance to that of the teacher model with faster inference time.
  arXiv  Detail & Related papers  (2020-12-08T11:09:22Z)
• Learning to Reweight with Deep Interactions [104.68509759134878]
  We propose an improved data reweighting algorithm, in which the student model provides its internal states to the teacher model. Experiments on image classification with clean/noisy labels and neural machine translation empirically demonstrate that our algorithm makes significant improvement over previous methods.
  arXiv  Detail & Related papers  (2020-07-09T09:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.