Blending Reward Functions via Few Expert Demonstrations for Faithful and Accurate Knowledge-Grounded Dialogue Generation

• URL: http://arxiv.org/abs/2311.00953v1
• Date: Thu, 2 Nov 2023 02:42:41 GMT
• Title: Blending Reward Functions via Few Expert Demonstrations for Faithful and Accurate Knowledge-Grounded Dialogue Generation
• Authors: Wanyu Du, Yangfeng Ji
• Abstract summary: We leverage reinforcement learning algorithms to overcome the above challenges by introducing a novel reward function. Our reward function combines an accuracy metric and a faithfulness metric to provide a balanced quality judgment of generated responses.
• Score: 22.38338205905379
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The development of trustworthy conversational information-seeking systems relies on dialogue models that can generate faithful and accurate responses based on relevant knowledge texts. However, two main challenges hinder this task. Firstly, language models may generate hallucinations due to data biases present in their pretraining corpus. Secondly, knowledge texts often contain redundant and irrelevant information that distracts the model's attention from the relevant text span. Previous works use additional data annotations on the knowledge texts to learn a knowledge identification module in order to bypass irrelevant information, but collecting such high-quality span annotations can be costly. In this work, we leverage reinforcement learning algorithms to overcome the above challenges by introducing a novel reward function. Our reward function combines an accuracy metric and a faithfulness metric to provide a balanced quality judgment of generated responses, which can be used as a cost-effective approximation to a human preference reward model when only a few preference annotations are available. Empirical experiments on two conversational information-seeking datasets demonstrate that our method can compete with other strong supervised learning baselines.

Related papers

• Maintaining Informative Coherence: Migrating Hallucinations in Large Language Models via Absorbing Markov Chains [6.920249042435973]
  Large Language Models (LLMs) are powerful tools for text generation, translation, and summarization. LLMs often suffer from hallucinations-instances where they fail to maintain the fidelity and coherence of contextual information. We propose a novel decoding strategy that leverages absorbing Markov chains to quantify the significance of contextual information.
  arXiv  Detail & Related papers  (2024-10-27T04:51:18Z)
• Large Language Model Augmented Exercise Retrieval for Personalized Language Learning [2.946562343070891]
  We find that vector similarity approaches poorly capture the relationship between exercise content and the language that learners use to express what they want to learn. We leverage the generative capabilities of large language models to bridge the gap by synthesizing hypothetical exercises based on the learner's input. Our approach, which we call mHyER, overcomes three challenges: (1) lack of relevance labels for training, (2) unrestricted learner input content, and (3) low semantic similarity between input and retrieval candidates.
  arXiv  Detail & Related papers  (2024-02-08T20:35:31Z)
• DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text [73.68051228972024]
  Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge. Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
  arXiv  Detail & Related papers  (2023-10-31T04:37:57Z)
• The Short Text Matching Model Enhanced with Knowledge via Contrastive Learning [8.350445155753167]
  This paper proposes a short Text Matching model that combines contrastive learning and external knowledge. To avoid noise, we use keywords as the main semantics of the original sentence to retrieve corresponding knowledge words in the knowledge base. Our designed model achieves state-of-the-art performance on two publicly available Chinese Text Matching datasets.
  arXiv  Detail & Related papers  (2023-04-08T03:24:05Z)
• Context-faithful Prompting for Large Language Models [51.194410884263135]
  Large language models (LLMs) encode parametric knowledge about world facts. Their reliance on parametric knowledge may cause them to overlook contextual cues, leading to incorrect predictions in context-sensitive NLP tasks. We assess and enhance LLMs' contextual faithfulness in two aspects: knowledge conflict and prediction with abstention.
  arXiv  Detail & Related papers  (2023-03-20T17:54:58Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfer [61.34424171458634]
  We study whether integrating visual knowledge into a language model can fill the gap. Our experiments show that visual knowledge transfer can improve performance in both low-resource and fully supervised settings.
  arXiv  Detail & Related papers  (2022-03-14T22:02:40Z)
• Knowledge Enhanced Fine-Tuning for Better Handling Unseen Entities in Dialogue Generation [33.806361531386685]
  We introduce two auxiliary training objectives: 1) Interpret Masked Word, which conjectures the meaning of the masked entity given the context; 2) Hypernym Generation, which predicts the hypernym of the entity based on the context. Experiment results on two dialogue corpus verify the effectiveness of our methods under both knowledge available and unavailable settings.
  arXiv  Detail & Related papers  (2021-09-12T11:13:19Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)
• Low-Resource Knowledge-Grounded Dialogue Generation [74.09352261943913]
  We consider knowledge-grounded dialogue generation under a natural assumption that only limited training examples are available. We devise a disentangled response decoder in order to isolate parameters that depend on knowledge-grounded dialogues from the entire generation model. With only 1/8 training data, our model can achieve the state-of-the-art performance and generalize well on out-of-domain knowledge.
  arXiv  Detail & Related papers  (2020-02-24T16:20:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.