WebGPT: Browser-assisted question-answering with human feedback

• URL: http://arxiv.org/abs/2112.09332v1
• Date: Fri, 17 Dec 2021 05:43:43 GMT
• Title: WebGPT: Browser-assisted question-answering with human feedback
• Authors: Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman
• Abstract summary: We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.
• Score: 12.865185980752733
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the web. By setting up the task so that it can be performed by humans, we are able to train models on the task using imitation learning, and then optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. We train and evaluate our models on ELI5, a dataset of questions asked by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.

Related papers

• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models [115.501751261878]
  Fine-tuning language models(LMs) on human-generated data remains a prevalent practice. We investigate whether we can go beyond human data on tasks where we have access to scalar feedback. We find that ReST$EM$ scales favorably with model size and significantly surpasses fine-tuning only on human data.
  arXiv  Detail & Related papers  (2023-12-11T18:17:43Z)
• UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
  We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
  arXiv  Detail & Related papers  (2023-10-02T17:40:01Z)
• Overinformative Question Answering by Humans and Machines [26.31070412632125]
  We show that overinformativeness in human answering is driven by considerations of relevance to the questioner's goals. We show that GPT-3 is highly sensitive to the form of the prompt and only human-like answer patterns when guided by an example and cognitively-motivated explanation.
  arXiv  Detail & Related papers  (2023-05-11T21:41:41Z)
• Aligning Text-to-Image Models using Human Feedback [104.76638092169604]
  Current text-to-image models often generate images that are inadequately aligned with text prompts. We propose a fine-tuning method for aligning such models using human feedback. Our results demonstrate the potential for learning from human feedback to significantly improve text-to-image models.
  arXiv  Detail & Related papers  (2023-02-23T17:34:53Z)
• Elaboration-Generating Commonsense Question Answering at Scale [77.96137534751445]
  In question answering requiring common sense, language models (e.g., GPT-3) have been used to generate text expressing background knowledge. We finetune smaller language models to generate useful intermediate context, referred to here as elaborations. Our framework alternates between updating two language models -- an elaboration generator and an answer predictor -- allowing each to influence the other.
  arXiv  Detail & Related papers  (2022-09-02T18:32:09Z)
• Training Language Models with Natural Language Feedback [51.36137482891037]
  We learn from language feedback on model outputs using a three-step learning algorithm. In synthetic experiments, we first evaluate whether language models accurately incorporate feedback to produce refinements. Using only 100 samples of human-written feedback, our learning algorithm finetunes a GPT-3 model to roughly human-level summarization.
  arXiv  Detail & Related papers  (2022-04-29T15:06:58Z)
• Pre-Trained Neural Language Models for Automatic Mobile App User Feedback Answer Generation [9.105367401167129]
  Studies show that developers' answers to the mobile app users' feedbacks on app stores can increase the apps' star rating. To help app developers generate answers that are related to the users' issues, recent studies develop models to generate the answers automatically. In this paper, we evaluate pre-trained neural language models (PTMs) to generate replies to the mobile app user feedbacks.
  arXiv  Detail & Related papers  (2022-02-04T18:26:55Z)
• When in Doubt, Ask: Generating Answerable and Unanswerable Questions, Unsupervised [0.0]
  Question Answering (QA) is key for making possible a robust communication between human and machine. Modern language models used for QA have surpassed the human-performance in several essential tasks. This paper studies augmenting human-made datasets with synthetic data as a way of surmounting this problem.
  arXiv  Detail & Related papers  (2020-10-04T15:56:44Z)
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data [52.12342165926226]
  We leverage social media feedback data to build a large-scale training dataset for feedback prediction. We trained DialogRPT, a set of GPT-2 based models on 133M pairs of human feedback data. Our ranker outperforms the conventional dialog perplexity baseline with a large margin on predicting Reddit feedback.
  arXiv  Detail & Related papers  (2020-09-15T10:50:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.