Language Instructed Reinforcement Learning for Human-AI Coordination

• URL: http://arxiv.org/abs/2304.07297v2
• Date: Sat, 10 Jun 2023 20:00:45 GMT
• Title: Language Instructed Reinforcement Learning for Human-AI Coordination
• Authors: Hengyuan Hu, Dorsa Sadigh
• Abstract summary: We propose a novel framework, instructRL, that enables humans to specify what kind of strategies they expect from their AI partners through natural language instructions. We show that instructRL converges to human-like policies that satisfy the given instructions in a proof-of-concept environment and the challenging Hanabi benchmark.
• Score: 23.694362407434753
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: One of the fundamental quests of AI is to produce agents that coordinate well with humans. This problem is challenging, especially in domains that lack high quality human behavioral data, because multi-agent reinforcement learning (RL) often converges to different equilibria from the ones that humans prefer. We propose a novel framework, instructRL, that enables humans to specify what kind of strategies they expect from their AI partners through natural language instructions. We use pretrained large language models to generate a prior policy conditioned on the human instruction and use the prior to regularize the RL objective. This leads to the RL agent converging to equilibria that are aligned with human preferences. We show that instructRL converges to human-like policies that satisfy the given instructions in a proof-of-concept environment as well as the challenging Hanabi benchmark. Finally, we show that knowing the language instruction significantly boosts human-AI coordination performance in human evaluations in Hanabi.

Related papers

• Aligning Large Language Models from Self-Reference AI Feedback with one General Principle [61.105703857868775]
  We propose a self-reference-based AI feedback framework that enables a 13B Llama2-Chat to provide high-quality feedback. Specifically, we allow the AI to first respond to the user's instructions, then generate criticism of other answers based on its own response as a reference. Finally, we determine which answer better fits human preferences according to the criticism.
  arXiv  Detail & Related papers  (2024-06-17T03:51:46Z)
• Policy Learning with a Language Bottleneck [65.99843627646018]
  Policy Learning with a Language Bottleneck (PLLBB) is a framework enabling AI agents to generate linguistic rules. PLLBB alternates between a rule generation step guided by language models, and an update step where agents learn new policies guided by rules. In a two-player communication game, a maze solving task, and two image reconstruction tasks, we show thatPLLBB agents are not only able to learn more interpretable and generalizable behaviors, but can also share the learned rules with human users.
  arXiv  Detail & Related papers  (2024-05-07T08:40:21Z)
• Efficient Human-AI Coordination via Preparatory Language-based Convention [17.840956842806975]
  Existing methods for human-AI coordination typically train an agent to coordinate with a diverse set of policies or with human models fitted from real human data. We propose employing the large language model (LLM) to develop an action plan that effectively guides both human and AI. Our method achieves better alignment with human preferences and an average performance improvement of 15% compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-11-01T10:18:23Z)
• Improving Generalization of Alignment with Human Preferences through Group Invariant Learning [56.19242260613749]
  Reinforcement Learning from Human Feedback (RLHF) enables the generation of responses more aligned with human preferences. Previous work shows that Reinforcement Learning (RL) often exploits shortcuts to attain high rewards and overlooks challenging samples. We propose a novel approach that can learn a consistent policy via RL across various data groups or domains.
  arXiv  Detail & Related papers  (2023-10-18T13:54:15Z)
• Exploration with Principles for Diverse AI Supervision [88.61687950039662]
  Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI. While this generative AI approach has produced impressive results, it heavily leans on human supervision. This strong reliance on human oversight poses a significant hurdle to the advancement of AI innovation. We propose a novel paradigm termed Exploratory AI (EAI) aimed at autonomously generating high-quality training data.
  arXiv  Detail & Related papers  (2023-10-13T07:03:39Z)
• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision [84.31474052176343]
  Recent AI-assistant agents, such as ChatGPT, rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback to align the output with human intentions. This dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision. We propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision.
  arXiv  Detail & Related papers  (2023-05-04T17:59:28Z)
• Learning Complementary Policies for Human-AI Teams [22.13683008398939]
  We propose a framework for a novel human-AI collaboration for selecting advantageous course of action. Our solution aims to exploit the human-AI complementarity to maximize decision rewards.
  arXiv  Detail & Related papers  (2023-02-06T17:22:18Z)
• Human-AI Coordination via Human-Regularized Search and Learning [33.95649252941375]
  We develop a three-step algorithm that achieve strong performance in coordinating with real humans in the Hanabi benchmark. We first use a regularized search algorithm and behavioral cloning to produce a better human model that captures diverse skill levels. We show that our method beats a vanilla best response to behavioral cloning baseline by having experts play repeatedly with the two agents.
  arXiv  Detail & Related papers  (2022-10-11T03:46:12Z)
• Instructive artificial intelligence (AI) for human training, assistance, and explainability [0.24629531282150877]
  We show how a neural network might instruct human trainees as an alternative to traditional approaches to explainable AI (XAI) An AI examines human actions and calculates variations on the human strategy that lead to better performance. Results will be presented on AI instruction's ability to improve human decision-making and human-AI teaming in Hanabi.
  arXiv  Detail & Related papers  (2021-11-02T16:46:46Z)
• Weak Human Preference Supervision For Deep Reinforcement Learning [48.03929962249475]
  The current reward learning from human preferences could be used to resolve complex reinforcement learning (RL) tasks without access to a reward function. We propose a weak human preference supervision framework, for which we developed a human preference scaling model. Our established human-demonstration estimator requires human feedback only for less than 0.01% of the agent's interactions with the environment.
  arXiv  Detail & Related papers  (2020-07-25T10:37:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.