KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation

• URL: http://arxiv.org/abs/2306.15430v1
• Date: Tue, 27 Jun 2023 12:38:49 GMT
• Title: KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation
• Authors: Jiaqi Bai, Zhao Yan, Jian Yang, Xinnian Liang, Hongcheng Guo, Zhoujun Li
• Abstract summary: Existing knowledge-grounded conversation systems generate responses typically in a retrieve-then-generate manner. We propose a two-stage tuning framework, bypassing the retrieval process by injecting prior knowledge into the lightweight knowledge prefix. KnowPrefix-Tuning outperforms fine-tuning and other lightweight tuning approaches.
• Score: 37.36605012674462
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing knowledge-grounded conversation systems generate responses typically in a retrieve-then-generate manner. They require a large knowledge base and a strong knowledge retrieval component, which is time- and resource-consuming. In this paper, we address the challenge by leveraging the inherent knowledge encoded in the pre-trained language models (PLMs). We propose Knowledgeable Prefix Tuning (KnowPrefix-Tuning), a two-stage tuning framework, bypassing the retrieval process in a knowledge-grounded conversation system by injecting prior knowledge into the lightweight knowledge prefix. The knowledge prefix is a sequence of continuous knowledge-specific vectors that can be learned during training. In addition, we propose a novel interactive re-parameterization mechanism that allows the prefix to interact fully with the PLM during the optimization of response generation. Experimental results demonstrate that KnowPrefix-Tuning outperforms fine-tuning and other lightweight tuning approaches, and performs comparably with strong retrieval-based baselines while being $3\times$ faster during inference.

Related papers

• TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models [31.209774088374374]
  This paper introduces TRELM, a Robust and Efficient Pre-training framework for Knowledge-Enhanced Language Models. We employ a robust approach to inject knowledge triples and employ a knowledge-augmented memory bank to capture valuable information. We show that TRELM reduces pre-training time by at least 50% and outperforms other KEPLMs in knowledge probing tasks and multiple knowledge-aware language understanding tasks.
  arXiv  Detail & Related papers  (2024-03-17T13:04:35Z)
• Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment [77.56326872997407]
  Pretrained language models (PLMs) based knowledge-grounded dialogue systems are prone to generate responses that are factually inconsistent with the provided knowledge source. Inspired by previous work which identified that feed-forward networks (FFNs) within Transformers are responsible for factual knowledge expressions, we investigate two methods to efficiently improve the factual expression capability.
  arXiv  Detail & Related papers  (2023-10-12T14:44:05Z)
• UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models [100.4659557650775]
  We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:33:28Z)
• Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning [113.58691755215663]
  We develop RetroPrompt to help a model strike a balance between generalization and memorization. In contrast with vanilla prompt learning, RetroPrompt constructs an open-book knowledge-store from training instances. Extensive experiments demonstrate that RetroPrompt can obtain better performance in both few-shot and zero-shot settings.
  arXiv  Detail & Related papers  (2022-05-29T16:07:30Z)
• Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters [52.725200145600624]
  We propose KnowExpert to bypass the retrieval process by injecting prior knowledge into the pre-trained language models with lightweight adapters. Experimental results show that KnowExpert performs comparably with the retrieval-based baselines.
  arXiv  Detail & Related papers  (2021-05-13T12:33:23Z)
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction [111.74812895391672]
  We propose a Knowledge-aware Prompt-tuning approach with synergistic optimization (KnowPrompt) We inject latent knowledge contained in relation labels into prompt construction with learnable virtual type words and answer words.
  arXiv  Detail & Related papers  (2021-04-15T17:57:43Z)
• K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining [5.178964604577459]
  We focus on improving model pretraining by leveraging explicit knowledge. To be specific, we first match knowledge facts from knowledge graph (KG) and then add a knowledge injunction layer to transformer directly. The experimental results show that solely by adding external knowledge to transformer can improve the learning performance on many NLP tasks.
  arXiv  Detail & Related papers  (2021-03-25T06:14:18Z)
• REALM: Retrieval-Augmented Language Model Pre-Training [37.3178586179607]
  We augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA)
  arXiv  Detail & Related papers  (2020-02-10T18:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.