When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels

• URL: http://arxiv.org/abs/2210.15893v1
• Date: Fri, 28 Oct 2022 04:57:21 GMT
• Title: When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels
• Authors: Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, Jing Xu
• Abstract summary: Juicer is a framework to make use of both binary and free-form textual human feedback. We find that augmenting training with model-corrected replies improves the final dialogue model.
• Score: 34.6235464256814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.

Related papers

• Improving the Validity of Automatically Generated Feedback via Reinforcement Learning [50.067342343957876]
  We propose a framework for feedback generation that optimize both correctness and alignment using reinforcement learning (RL) Specifically, we use GPT-4's annotations to create preferences over feedback pairs in an augmented dataset for training via direct preference optimization (DPO)
  arXiv  Detail & Related papers  (2024-03-02T20:25:50Z)
• RefuteBench: Evaluating Refuting Instruction-Following for Large Language Models [17.782410287625645]
  This paper proposes a benchmark, RefuteBench, covering tasks such as question answering, machine translation, and email writing. The evaluation aims to assess whether models can positively accept feedback in form of refuting instructions and whether they can consistently adhere to user demands throughout the conversation.
  arXiv  Detail & Related papers  (2024-02-21T01:39:56Z)
• Leveraging Implicit Feedback from Deployment Data in Dialogue [83.02878726357523]
  We study improving social conversational agents by learning from natural dialogue between users and a deployed model. We leverage signals like user response length, sentiment and reaction of the future human utterances in the collected dialogue episodes.
  arXiv  Detail & Related papers  (2023-07-26T11:34:53Z)
• Fine-Grained Human Feedback Gives Better Rewards for Language Model Training [108.25635150124539]
  Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects.
  arXiv  Detail & Related papers  (2023-06-02T17:11:37Z)
• WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue [17.663449579168297]
  We simulate a dialogue between an agent and a user (modelled similar to an agent with supervised learning objective) to interact with each other. The agent uses dynamic blocking to generate ranked diverse responses and exploration-exploitation to select among the Top-K responses. Empirical studies with two benchmarks indicate that our model can significantly out-perform the response quality and lead to a successful conversation.
  arXiv  Detail & Related papers  (2021-08-01T08:00:45Z)
• Improving Response Quality with Backward Reasoning in Open-domain Dialogue Systems [53.160025961101354]
  We propose to train the generation model in a bidirectional manner by adding a backward reasoning step to the vanilla encoder-decoder training. The proposed backward reasoning step pushes the model to produce more informative and coherent content. Our method can improve response quality without introducing side information.
  arXiv  Detail & Related papers  (2021-04-30T20:38:27Z)
• Learning Improvised Chatbots from Adversarial Modifications of Natural Language Feedback [19.026954124876582]
  We propose a generative adversarial model that converts noisy feedback into a plausible natural response in a conversation. The generator's goal is to convert the feedback into a response that answers the user's previous utterance and to fool the discriminator.
  arXiv  Detail & Related papers  (2020-10-14T17:33:37Z)
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data [52.12342165926226]
  We leverage social media feedback data to build a large-scale training dataset for feedback prediction. We trained DialogRPT, a set of GPT-2 based models on 133M pairs of human feedback data. Our ranker outperforms the conventional dialog perplexity baseline with a large margin on predicting Reddit feedback.
  arXiv  Detail & Related papers  (2020-09-15T10:50:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.