Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding

• URL: http://arxiv.org/abs/2309.15028v3
• Date: Tue, 2 Apr 2024 17:51:49 GMT
• Title: Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding
• Abstract summary: Inference-time search algorithms may seem unnecessary when generating natural language text based on Proximal Policy Optimization (PPO) In this paper, we demonstrate that it is possible to get extra mileage out of PPO by integrating Monte-Carlo Tree Search (MCTS) on top. We present PPO-MCTS, which can integrate the value network from PPO to work closely with the policy network during inference-time generation.
• Score: 104.77043794433777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inference-time search algorithms such as Monte-Carlo Tree Search (MCTS) may seem unnecessary when generating natural language text based on state-of-the-art reinforcement learning such as Proximal Policy Optimization (PPO). In this paper, we demonstrate that it is possible to get extra mileage out of PPO by integrating MCTS on top. The key idea is not to throw out the value network, a byproduct of PPO training for evaluating partial output sequences, when decoding text out of the policy network. More concretely, we present a novel value-guided decoding algorithm called PPO-MCTS, which can integrate the value network from PPO to work closely with the policy network during inference-time generation. Compared to prior approaches based on MCTS for controlled text generation, the key strength of our approach is to reduce the fundamental mismatch of the scoring mechanisms of the partial outputs between training and test. Evaluation on four text generation tasks demonstrate that PPO-MCTS greatly improves the preferability of generated text compared to the standard practice of using only the PPO policy. Our results demonstrate the promise of search algorithms even on top of the aligned language models from PPO, and the under-explored benefit of the value network.

Related papers

• SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks [41.49967840381499]
  Sequence-Level PPO (SPPO) is a scalable algorithm that harmonizes the sampletextual efficiency of PPO with the stability of outcome-based updates.<n>SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods.
  arXiv  Detail & Related papers  (2026-04-10T01:58:21Z)
• Owen-Shapley Policy Optimization (OSPO): A Principled RL Algorithm for Generative Search LLMs [12.75200353208858]
  Owen-Shapley Policy Optimization (OSPO) is a framework that redistributes sequence-level advantages based on tokens' marginal contributions to outcomes.<n>Unlike value-model-based methods requiring additional computation, OSPO employs potential-based reward shaping via Shapley-Owen attributions to assign segment-level credit.<n> Experiments on Amazon ESCI and H&M Fashion datasets show consistent gains over baselines.
  arXiv  Detail & Related papers  (2026-01-13T10:17:46Z)
• TD-M(PC)$^2$: Improving Temporal Difference MPC Through Policy Constraint [11.347808936693152]
  Model-based reinforcement learning algorithms combine model-based planning and learned value/policy prior. Existing methods that rely on standard SAC-style policy iteration for value learning often result in emphpersistent value overestimation. We propose a policy regularization term reducing out-of-distribution (OOD) queries, thereby improving value learning.
  arXiv  Detail & Related papers  (2025-02-05T19:08:42Z)
• Token-level Proximal Policy Optimization for Query Generation [45.81132350185301]
  State-of-the-art query generation methods leverage Large Language Models (LLMs) for their strong capabilities in context understanding and text generation. We propose Token-level Proximal Policy Optimization (TPPO), a noval approach designed to empower LLMs perform better in query generation through fine-tuning. TPPO is based on the Reinforcement Learning from AI Feedback (RLAIF) paradigm, consisting of a token-level reward model and a token-level proximal policy optimization module.
  arXiv  Detail & Related papers  (2024-11-01T16:36:14Z)
• VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment [66.80143024475635]
  We propose VinePPO, a straightforward approach to compute unbiased Monte Carlo-based estimates. We show that VinePPO consistently outperforms PPO and other RL-free baselines across MATH and GSM8K datasets.
  arXiv  Detail & Related papers  (2024-10-02T15:49:30Z)
• DPO Meets PPO: Reinforced Token Optimization for RLHF [36.97894955691627]
  We introduce a framework that models RLHF problems as a Markov decision process (MDP) Under this framework, we introduce an algorithm, dubbed as Reinforced Token Optimization (textttRTO), which learns the token-wise reward function from preference data. For its practical implementation, textttRTO innovatively integrates Direct Preference Optimization (DPO) and Proximal Policy Optimization.
  arXiv  Detail & Related papers  (2024-04-29T17:58:30Z)
• From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function [50.812404038684505]
  We show that we can derive DPO in the token-level MDP as a general inverse Q-learning algorithm, which satisfies the Bellman equation. We discuss applications of our work, including information elicitation in multi-turn dialogue, reasoning, agentic applications and end-to-end training of multi-model systems.
  arXiv  Detail & Related papers  (2024-04-18T17:37:02Z)
• Learning Logic Specifications for Soft Policy Guidance in POMCP [71.69251176275638]
  Partially Observable Monte Carlo Planning (POMCP) is an efficient solver for Partially Observable Markov Decision Processes (POMDPs) POMCP suffers from sparse reward function, namely, rewards achieved only when the final goal is reached. In this paper, we use inductive logic programming to learn logic specifications from traces of POMCP executions.
  arXiv  Detail & Related papers  (2023-03-16T09:37:10Z)
• Exploiting the Textual Potential from Vision-Language Pre-training for Text-based Person Search [17.360982091304137]
  Text-based Person Search (TPS) is targeted on retrieving pedestrians to match text descriptions instead of query images. Recent Vision-Language Pre-training models can bring transferable knowledge to downstream TPS tasks, resulting in more efficient performance gains. However, existing TPS methods only utilize pre-trained visual encoders, neglecting the corresponding textual representation.
  arXiv  Detail & Related papers  (2023-03-08T10:41:22Z)
• Turning a CLIP Model into a Scene Text Detector [56.86413150091367]
  Recently, pretraining approaches based on vision language models have made effective progresses in the field of text detection. This paper proposes a new method, termed TCM, focusing on Turning the CLIP Model directly for text detection without pretraining process.
  arXiv  Detail & Related papers  (2023-02-28T06:06:12Z)
• You May Not Need Ratio Clipping in PPO [117.03368180633463]
  Proximal Policy Optimization (PPO) methods learn a policy by iteratively performing multiple mini-batch optimization epochs of a surrogate objective with one set of sampled data. Ratio clipping PPO is a popular variant that clips the probability ratios between the target policy and the policy used to collect samples. We show in this paper that such ratio clipping may not be a good option as it can fail to effectively bound the ratios. We show that ESPO can be easily scaled up to distributed training with many workers, delivering strong performance as well.
  arXiv  Detail & Related papers  (2022-01-31T20:26:56Z)
• Text Generation with Efficient (Soft) Q-Learning [91.47743595382758]
  Reinforcement learning (RL) offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. We introduce a new RL formulation for text generation from the soft Q-learning perspective. We apply the approach to a wide range of tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
  arXiv  Detail & Related papers  (2021-06-14T18:48:40Z)
• Neural Machine Translation with Monte-Carlo Tree Search [3.8073142980733]
  Recent algorithms in machine translation have included a value network to assist the policy network when deciding which word to output at each step of the translation. The main idea of our project is to instead leverage Monte-Carlo Tree Search (MCTS) to search for good output words with guidance from a combined policy and value network architecture. Experiments using the IWLST14 German to English translation dataset show that our method outperforms the actor-critic methods used in recent machine translation papers.
  arXiv  Detail & Related papers  (2020-04-27T01:03:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.