Pretraining Language Models with Human Preferences

• URL: http://arxiv.org/abs/2302.08582v2
• Date: Wed, 14 Jun 2023 13:27:58 GMT
• Title: Pretraining Language Models with Human Preferences
• Authors: Tomasz Korbak and Kejian Shi and Angelica Chen and Rasika Bhalerao and Christopher L. Buckley and Jason Phang and Samuel R. Bowman and Ethan Perez
• Abstract summary: Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM. Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences.
• Score: 21.724817280998696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM: falsehoods, offensive comments, personally identifiable information, low-quality or buggy code, and more. Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences. We benchmark five objectives for pretraining with human feedback across three tasks and study how they affect the trade-off between alignment and capabilities of pretrained LMs. We find a Pareto-optimal and simple approach among those we explored: conditional training, or learning distribution over tokens conditional on their human preference scores given by a reward model. Conditional training reduces the rate of undesirable content by up to an order of magnitude, both when generating without a prompt and with an adversarially-chosen prompt. Moreover, conditional training maintains the downstream task performance of standard LM pretraining, both before and after task-specific finetuning. Pretraining with human feedback results in much better preference satisfaction than standard LM pretraining followed by finetuning with feedback, i.e., learning and then unlearning undesirable behavior. Our results suggest that we should move beyond imitation learning when pretraining LMs and incorporate human preferences from the start of training.

Related papers

• Instruction Pre-Training: Language Models are Supervised Multitask Learners [115.95022434390181]
  In this paper, we propose a framework that augments massive raw corpora with instruction-response pairs to pre-train language models (LMs) In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of Instruction Pre-Training.
  arXiv  Detail & Related papers  (2024-06-20T16:55:33Z)
• Aligning language models with human preferences [5.0994393083677]
  Language models (LMs) trained on vast quantities of text data can acquire sophisticated skills. They also manifest behaviors that violate human preferences. I explore several approaches to aligning LMs with human preferences.
  arXiv  Detail & Related papers  (2024-04-18T12:55:18Z)
• Preference-grounded Token-level Guidance for Language Model Fine-tuning [105.88789610320426]
  Aligning language models with preferences is an important problem in natural language generation. For LM training, based on the amount of supervised data, we present two *minimalist* learning objectives that utilize the learned guidance. In experiments, our method performs competitively on two distinct representative LM tasks.
  arXiv  Detail & Related papers  (2023-06-01T07:00:07Z)
• Training Language Models with Language Feedback at Scale [50.70091340506957]
  We introduce learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback.
  arXiv  Detail & Related papers  (2023-03-28T17:04:15Z)
• Improving Code Generation by Training with Natural Language Feedback [69.52985513422381]
  We formalize an algorithm for learning from natural language feedback at training time instead, which we call learning from Language Feedback (ILF) ILF requires only a small amount of human-written feedback during training and does not require the same feedback at test time, making it both user-friendly and sample-efficient. We use ILF to improve a Codegen-Mono 6.1B model's pass@1 rate by 38% relative (and 10% absolute) on the Mostly Basic Python Problems (MBPP) benchmark.
  arXiv  Detail & Related papers  (2023-03-28T16:15:31Z)
• Meet in the Middle: A New Pre-training Paradigm [41.52858444519968]
  Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion. We propose a new pre-training paradigm with techniques that jointly improve the training data efficiency. We show the effectiveness of our pre-training paradigm with extensive experiments on both programming and natural language models.
  arXiv  Detail & Related papers  (2023-03-13T17:17:11Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning [80.24059713295165]
  The Logarithm of Maximum Evidence (LogME) can be used to assess pre-trained models for transfer learning. Compared to brute-force fine-tuning, LogME brings over $3000times$ speedup in wall-clock time.
  arXiv  Detail & Related papers  (2021-02-22T13:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.