Reinforcing Language Agents via Policy Optimization with Action Decomposition

• URL: http://arxiv.org/abs/2405.15821v1
• Date: Thu, 23 May 2024 14:01:44 GMT
• Title: Reinforcing Language Agents via Policy Optimization with Action Decomposition
• Authors: Muning Wen, Ziyu Wan, Weinan Zhang, Jun Wang, Ying Wen,
• Abstract summary: This paper proposes decomposing language agent optimization from the action level to the token level. We then derive the Bellman backup with Action Decomposition (BAD) to integrate credit assignments for both intra-action and inter-action tokens. Implementing BAD within the PPO algorithm, we introduce Policy Optimization with Action Decomposition (POAD)
• Score: 36.984163245259936
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models as intelligent agents push the boundaries of sequential decision-making agents but struggle with limited knowledge of environmental dynamics and exponentially huge action space. Recent efforts like GLAM and TWOSOME manually constrain the action space to a restricted subset and employ reinforcement learning to align agents' knowledge with specific environments. However, they overlook fine-grained credit assignments for intra-action tokens, which is essential for efficient language agent optimization, and rely on human's prior knowledge to restrict action space. This paper proposes decomposing language agent optimization from the action level to the token level, offering finer supervision for each intra-action token and manageable optimization complexity in environments with unrestricted action spaces. Beginning with the simplification of flattening all actions, we theoretically explore the discrepancies between action-level optimization and this naive token-level optimization. We then derive the Bellman backup with Action Decomposition (BAD) to integrate credit assignments for both intra-action and inter-action tokens, effectively eliminating the discrepancies. Implementing BAD within the PPO algorithm, we introduce Policy Optimization with Action Decomposition (POAD). POAD benefits from a finer-grained credit assignment process and lower optimization complexity, leading to enhanced learning efficiency and generalization abilities in aligning language agents with interactive environments. We validate POAD across diverse testbeds, with results affirming the advantages of our approach and the correctness of our theoretical analysis.

Related papers

• Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer [52.09480867526656]
  We identify the source of misalignment as a form of distributional shift and uncertainty in learning human preferences. To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model. Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines a preference optimization loss and a supervised learning loss.
  arXiv  Detail & Related papers  (2024-05-26T05:38:50Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Efficient Reinforcement Learning via Decoupling Exploration and Utilization [6.305976803910899]
  Reinforcement Learning (RL) has achieved remarkable success across multiple fields and applications, including gaming, robotics, and autonomous vehicles. In this work, our aim is to train agent with efficient learning by decoupling exploration and utilization, so that agent can escaping the conundrum of suboptimal Solutions. The above idea is implemented in the proposed OPARL (Optimistic and Pessimistic Actor Reinforcement Learning) algorithm.
  arXiv  Detail & Related papers  (2023-12-26T09:03:23Z)
• APS: Active Pretraining with Successor Features [96.24533716878055]
  We show that by reinterpreting and combining successorcitepHansenFast with non entropy, the intractable mutual information can be efficiently optimized. The proposed method Active Pretraining with Successor Feature (APS) explores the environment via non entropy, and the explored data can be efficiently leveraged to learn behavior.
  arXiv  Detail & Related papers  (2021-08-31T16:30:35Z)
• Attention Actor-Critic algorithm for Multi-Agent Constrained Co-operative Reinforcement Learning [3.296127938396392]
  We consider the problem of computing optimal actions for Reinforcement Learning (RL) agents in a co-operative setting. We extend this algorithm to the constrained multi-agent RL setting.
  arXiv  Detail & Related papers  (2021-01-07T03:21:15Z)
• One-shot Policy Elicitation via Semantic Reward Manipulation [2.668480521943575]
  We present Single-shot Policy Explanation for Augmenting Rewards (SPEAR), a novel sequential optimization algorithm. We show that SPEAR makes substantial improvements over the state-of-the-art in terms of runtime and addressable problem size.
  arXiv  Detail & Related papers  (2021-01-06T04:11:22Z)
• Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models [54.569004548170824]
  We show that careful masking strategies can bridge the knowledge gap of masked language models. We propose an effective training strategy by adversarially masking out those tokens which are harder to adversarial by the underlying.
  arXiv  Detail & Related papers  (2020-10-05T01:49:47Z)
• Discrete Action On-Policy Learning with Action-Value Critic [72.20609919995086]
  Reinforcement learning (RL) in discrete action space is ubiquitous in real-world applications, but its complexity grows exponentially with the action-space dimension. We construct a critic to estimate action-value functions, apply it on correlated actions, and combine these critic estimated action values to control the variance of gradient estimation. These efforts result in a new discrete action on-policy RL algorithm that empirically outperforms related on-policy algorithms relying on variance control techniques.
  arXiv  Detail & Related papers  (2020-02-10T04:23:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.