Improving scripts with a memory of natural feedback

• URL: http://arxiv.org/abs/2112.09737v1
• Date: Thu, 16 Dec 2021 07:01:28 GMT
• Title: Improving scripts with a memory of natural feedback
• Authors: Niket Tandon, Aman Madaan, Peter Clark, Yiming Yang
• Abstract summary: We create a dynamic memory architecture with a growing memory of feedbacks about errors in the output. On a script generation task, we show empirically that the model learns to apply feedback effectively. This is a first step towards strengthening deployed models, potentially broadening their utility.
• Score: 38.81097942561449
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How can an end-user provide feedback if a deployed structured prediction model generates incorrect output? Our goal is to allow users to correct errors directly through interaction, without retraining, by giving feedback on the model's output. We create a dynamic memory architecture with a growing memory of feedbacks about errors in the output. Given a new, unseen input, our model can use feedback from a similar, past erroneous state. On a script generation task, we show empirically that the model learns to apply feedback effectively (up to 30 points improvement), while avoiding similar past mistakes after deployment (up to 10 points improvement on an unseen set). This is a first step towards strengthening deployed models, potentially broadening their utility.

Related papers

• Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision [120.40788744292739]
  We propose a two-player paradigm that separates the roles of reasoning and critique models. We first propose AutoMathCritique, an automated and scalable framework for collecting critique data. We demonstrate that the critique models consistently improve the actor's performance on difficult queries at test-time.
  arXiv  Detail & Related papers  (2024-11-25T17:11:54Z)
• Interpret the Internal States of Recommendation Model with Sparse Autoencoder [26.021277330699963]
  RecSAE is an automatic, generalizable probing method for interpreting the internal states of Recommendation models. We train an autoencoder with sparsity constraints to reconstruct internal activations of recommendation models. We automated the construction of concept dictionaries based on the relationship between latent activations and input item sequences.
  arXiv  Detail & Related papers  (2024-11-09T08:22:31Z)
• RLVF: Learning from Verbal Feedback without Overgeneralization [94.19501420241188]
  We study the problem of incorporating verbal feedback without such overgeneralization. We develop a new method Contextualized Critiques with Constrained Preference Optimization (C3PO) Our approach effectively applies verbal feedback to relevant scenarios while preserving existing behaviors for other contexts.
  arXiv  Detail & Related papers  (2024-02-16T18:50:24Z)
• What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinement [38.93348195407474]
  Language models deployed in the wild make errors. Updating the model with the corrected error instances causes catastrophic forgetting. We propose a partially interpretable forecasting model based on the observation that changes in pre-softmax logit scores of pretraining examples resemble that of online learned examples.
  arXiv  Detail & Related papers  (2024-02-02T19:43:15Z)
• Training Language Models with Language Feedback at Scale [50.70091340506957]
  We introduce learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback.
  arXiv  Detail & Related papers  (2023-03-28T17:04:15Z)
• XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models [33.81019305179569]
  Explanation-based model debug aims to resolve spurious biases by showing human users explanations of model behavior. We propose XMD: the first open-source, end-to-end framework for explanation-based model debug. XMD automatically updates the model in real time, by regularizing the model so that its explanations align with the user feedback.
  arXiv  Detail & Related papers  (2022-10-30T23:09:09Z)
• Towards Teachable Reasoning Systems [29.59387051046722]
  We develop a teachable reasoning system for question-answering (QA) Our approach is three-fold: First, generated chains of reasoning show how answers are implied by the system's own internal beliefs. Second, users can interact with the explanations to identify erroneous model beliefs and provide corrections. Third, we augment the model with a dynamic memory of such corrections.
  arXiv  Detail & Related papers  (2022-04-27T17:15:07Z)
• Memory-assisted prompt editing to improve GPT-3 after deployment [55.62352349324132]
  We show how a (simulated) user can interactively teach a deployed GPT-3, doubling its accuracy on basic lexical tasks. Our simple idea is a first step towards strengthening deployed models, potentially broadening their utility.
  arXiv  Detail & Related papers  (2022-01-16T10:11:37Z)
• Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting [100.75479161884935]
  We propose a novel training paradigm called Remembering for the Right Reasons (RRR) RRR stores visual model explanations for each example in the buffer and ensures the model has "the right reasons" for its predictions. We demonstrate how RRR can be easily added to any memory or regularization-based approach and results in reduced forgetting.
  arXiv  Detail & Related papers  (2020-10-04T10:05:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.