Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model

• URL: http://arxiv.org/abs/2310.09520v4
• Date: Tue, 2 Jan 2024 00:04:13 GMT
• Title: Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
• Authors: Haikang Deng, Colin Raffel
• Abstract summary: Reward-Augmented Decoding (RAD) is a text generation procedure that uses a small unidirectional reward model to encourage a language model to generate text that has certain properties. By using a unidirectional reward model, RAD can cache activations from prior generation steps to decrease computational overhead.
• Score: 47.722856876213946
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While large language models have proven effective in a huge range of downstream applications, they often generate text that is problematic or lacks a desired attribute. In this paper, we introduce Reward-Augmented Decoding (RAD), a text generation procedure that uses a small unidirectional reward model to encourage a language model to generate text that has certain properties. Specifically, RAD uses the reward model to score generations as they are produced and rescales sampling probabilities to favor high-reward tokens. By using a unidirectional reward model, RAD can cache activations from prior generation steps to decrease computational overhead. Through experiments on generating non-toxic and sentiment-controlled text, we demonstrate that RAD performs best among methods that change only the generation procedure and matches the performance of state-of-the-art methods that involve re-training the language model. We further validate that RAD is effective on very large language models while incurring a minimal computational overhead.

Related papers

• Efficient Controlled Language Generation with Low-Rank Autoregressive Reward Models [13.38174941551702]
  We revisit the reward augmented decoding (RAD) approach to control the generation from a language model using the scores from a task-specific reward model. We show that RAD is designed to support high flexibility when representing the reward matrices, which leads to a higher computational costs during decoding. Motivated by this, we propose a simpler but more efficient low-rank parametrization of the reward model enabling fast and effective guided decoding.
  arXiv  Detail & Related papers  (2024-07-05T16:11:03Z)
• Embedded Named Entity Recognition using Probing Classifiers [10.573861741540853]
  EMBER enables streaming named entity recognition in decoder-only language models without fine-tuning them. We show that EMBER maintains high token generation rates, with only a negligible decrease in speed of around 1%. We make our code and data available online, including a toolkit for training, testing, and deploying efficient token classification models.
  arXiv  Detail & Related papers  (2024-03-18T12:58:16Z)
• Reinforcement Learning with Token-level Feedback for Controllable Text Generation [16.117006822479407]
  We propose a novel reinforcement learning algorithm named TOLE which formulates TOken-LEvel rewards for controllable text generation. Experimental results show that our algorithm can achieve superior performance on both single-attribute and multi-attribute control tasks.
  arXiv  Detail & Related papers  (2024-03-18T08:18:37Z)
• Controlled Text Generation via Language Model Arithmetic [7.687678490751105]
  We introduce model arithmetic, a novel inference framework for composing and biasing Large Language Models. We show that model arithmetic allows fine-grained control of generated text while outperforming state-of-the-art on the task of toxicity reduction.
  arXiv  Detail & Related papers  (2023-11-24T13:41:12Z)
• Extrapolating Multilingual Understanding Models as Multilingual Generators [82.1355802012414]
  This paper explores methods to empower multilingual understanding models the generation abilities to get a unified model. We propose a textbfSemantic-textbfGuided textbfAlignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters.
  arXiv  Detail & Related papers  (2023-05-22T15:33:21Z)
• Tractable Control for Autoregressive Language Generation [82.79160918147852]
  We propose to use tractable probabilistic models (TPMs) to impose lexical constraints in autoregressive text generation models. We show that GeLaTo achieves state-of-the-art performance on challenging benchmarks for constrained text generation. Our work opens up new avenues for controlling large language models and also motivates the development of more expressive TPMs.
  arXiv  Detail & Related papers  (2023-04-15T00:19:44Z)
• Twist Decoding: Diverse Generators Guide Each Other [116.20780037268801]
  We introduce Twist decoding, a simple and general inference algorithm that generates text while benefiting from diverse models. Our method does not assume the vocabulary, tokenization or even generation order is shared.
  arXiv  Detail & Related papers  (2022-05-19T01:27:53Z)
• Text Generation with Efficient (Soft) Q-Learning [91.47743595382758]
  Reinforcement learning (RL) offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. We introduce a new RL formulation for text generation from the soft Q-learning perspective. We apply the approach to a wide range of tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
  arXiv  Detail & Related papers  (2021-06-14T18:48:40Z)
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training [93.79766670391618]
  We present POINTER, a novel insertion-based approach for hard-constrained text generation. The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner. The resulting coarse-to-fine hierarchy makes the generation process intuitive and interpretable.
  arXiv  Detail & Related papers  (2020-05-01T18:11:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.