Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models

• URL: http://arxiv.org/abs/2401.07553v3
• Date: Wed, 15 May 2024 12:08:21 GMT
• Title: Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models
• Authors: Xingzhou Lou, Junge Zhang, Ziyan Wang, Kaiqi Huang, Yali Du,
• Abstract summary: Safe reinforcement learning (RL) agents accomplish given tasks while adhering to specific constraints. We propose to use pre-trained language models (LM) to facilitate RL agents' comprehension of natural language constraints. Our method enhances safe policy learning under a diverse set of human-derived free-form natural language constraints.
• Score: 36.44404825103045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Safe reinforcement learning (RL) agents accomplish given tasks while adhering to specific constraints. Employing constraints expressed via easily-understandable human language offers considerable potential for real-world applications due to its accessibility and non-reliance on domain expertise. Previous safe RL methods with natural language constraints typically adopt a recurrent neural network, which leads to limited capabilities when dealing with various forms of human language input. Furthermore, these methods often require a ground-truth cost function, necessitating domain expertise for the conversion of language constraints into a well-defined cost function that determines constraint violation. To address these issues, we proposes to use pre-trained language models (LM) to facilitate RL agents' comprehension of natural language constraints and allow them to infer costs for safe policy learning. Through the use of pre-trained LMs and the elimination of the need for a ground-truth cost, our method enhances safe policy learning under a diverse set of human-derived free-form natural language constraints. Experiments on grid-world navigation and robot control show that the proposed method can achieve strong performance while adhering to given constraints. The usage of pre-trained LMs allows our method to comprehend complicated constraints and learn safe policies without the need for ground-truth cost at any stage of training or evaluation. Extensive ablation studies are conducted to demonstrate the efficacy of each part of our method.

Related papers

• Safe Multi-agent Reinforcement Learning with Natural Language Constraints [49.01100552946231]
  The role of natural language constraints in Safe Multi-agent Reinforcement Learning (MARL) is crucial, yet often overlooked. We propose a novel approach named Safe Multi-agent Reinforcement Learning with Natural Language constraints (SMALL) Our method leverages fine-tuned language models to interpret and process free-form textual constraints, converting them into semantic embeddings. These embeddings are then integrated into the multi-agent policy learning process, enabling agents to learn policies that minimize constraint violations while optimizing rewards.
  arXiv  Detail & Related papers  (2024-05-30T12:57:35Z)
• No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement [59.37775534633868]
  We introduce a novel method called language arithmetic, which enables training-free post-processing. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes.
  arXiv  Detail & Related papers  (2024-04-24T08:52:40Z)
• Scalable Language Model with Generalized Continual Learning [58.700439919096155]
  The Joint Adaptive Re-ization (JARe) is integrated with Dynamic Task-related Knowledge Retrieval (DTKR) to enable adaptive adjustment of language models based on specific downstream tasks. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting.
  arXiv  Detail & Related papers  (2024-04-11T04:22:15Z)
• Natural Language Reinforcement Learning [25.165291680493844]
  We introduce Natural Language Reinforcement Learning (NLRL), which combines RL principles with natural language representation. Specifically, NLRL redefines RL concepts like task objectives, policy, value function, Bellman equation, and policy iteration in natural language space.
  arXiv  Detail & Related papers  (2024-02-11T11:03:04Z)
• Controlled Text Generation with Natural Language Instructions [74.88938055638636]
  InstructCTG is a controlled text generation framework that incorporates different constraints. We first extract the underlying constraints of natural texts through a combination of off-the-shelf NLP tools and simple verbalizes. By prepending natural language descriptions of the constraints and a few demonstrations, we fine-tune a pre-trained language model to incorporate various types of constraints.
  arXiv  Detail & Related papers  (2023-04-27T15:56:34Z)
• Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translation [14.176720914723127]
  Natural Language-conditioned reinforcement learning (RL) enables the agents to follow human instructions. Previous approaches generally implemented language-conditioned RL by providing human instructions in natural language (NL) and training a following policy. We develop an inside-out scheme for natural language-conditioned RL by developing a task language (TL) that is task-related and unique.
  arXiv  Detail & Related papers  (2023-02-18T15:49:09Z)
• Offline RL for Natural Language Generation with Implicit Language Q Learning [87.76695816348027]
  Large language models can be inconsistent when it comes to completing user specified tasks. We propose a novel RL method, that combines both the flexible utility framework of RL with the ability of supervised learning. In addition to empirically validating ILQL, we present a detailed empirical analysis situations where offline RL can be useful in natural language generation settings.
  arXiv  Detail & Related papers  (2022-06-05T18:38:42Z)
• Safe Reinforcement Learning with Natural Language Constraints [39.70152978025088]
  We propose learning to interpret natural language constraints for safe RL. HazardWorld is a new multi-task benchmark that requires an agent to optimize reward while not violating constraints specified in free-form text. We show that our method achieves higher rewards (up to 11x) and fewer constraint violations (by 1.8x) compared to existing approaches.
  arXiv  Detail & Related papers  (2020-10-11T03:41:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.