Learning to Cooperate with Humans using Generative Agents

• URL: http://arxiv.org/abs/2411.13934v1
• Date: Thu, 21 Nov 2024 08:36:17 GMT
• Title: Learning to Cooperate with Humans using Generative Agents
• Authors: Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S. Du, Natasha Jaques,
• Abstract summary: Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL) We show emphlearning a generative model of human partners can effectively address this issue. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents.
• Score: 40.605931138995714
• License:
• Abstract: Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL). Current algorithms focus on training simulated human partner policies which are then used to train a Cooperator agent. The simulated human is produced either through behavior cloning over a dataset of human cooperation behavior, or by using MARL to create a population of simulated agents. However, these approaches often struggle to produce a Cooperator that can coordinate well with real humans, since the simulated humans fail to cover the diverse strategies and styles employed by people in the real world. We show \emph{learning a generative model of human partners} can effectively address this issue. Our model learns a latent variable representation of the human that can be regarded as encoding the human's unique strategy, intention, experience, or style. This generative model can be flexibly trained from any (human or neural policy) agent interaction data. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents. We evaluate our method -- \textbf{G}enerative \textbf{A}gent \textbf{M}odeling for \textbf{M}ulti-agent \textbf{A}daptation (GAMMA) -- on Overcooked, a challenging cooperative cooking game that has become a standard benchmark for zero-shot coordination. We conduct an evaluation with real human teammates, and the results show that GAMMA consistently improves performance, whether the generative model is trained on simulated populations or human datasets. Further, we propose a method for posterior sampling from the generative model that is biased towards the human data, enabling us to efficiently improve performance with only a small amount of expensive human interaction data.

Related papers

• Large Language Model-based Human-Agent Collaboration for Complex Task Solving [94.3914058341565]
  We introduce the problem of Large Language Models (LLMs)-based human-agent collaboration for complex task-solving. We propose a Reinforcement Learning-based Human-Agent Collaboration method, ReHAC. This approach includes a policy model designed to determine the most opportune stages for human intervention within the task-solving process.
  arXiv  Detail & Related papers  (2024-02-20T11:03:36Z)
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models [115.501751261878]
  Fine-tuning language models(LMs) on human-generated data remains a prevalent practice. We investigate whether we can go beyond human data on tasks where we have access to scalar feedback. We find that ReST$EM$ scales favorably with model size and significantly surpasses fine-tuning only on human data.
  arXiv  Detail & Related papers  (2023-12-11T18:17:43Z)
• Learning Human Action Recognition Representations Without Real Humans [66.61527869763819]
  We present a benchmark that leverages real-world videos with humans removed and synthetic data containing virtual humans to pre-train a model. We then evaluate the transferability of the representation learned on this data to a diverse set of downstream action recognition benchmarks. Our approach outperforms previous baselines by up to 5%.
  arXiv  Detail & Related papers  (2023-11-10T18:38:14Z)
• PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination [52.991211077362586]
  We propose a policy ensemble method to increase the diversity of partners in the population. We then develop a context-aware method enabling the ego agent to analyze and identify the partner's potential policy primitives. In this way, the ego agent is able to learn more universal cooperative behaviors for collaborating with diverse partners.
  arXiv  Detail & Related papers  (2023-01-16T12:14:58Z)
• Optimal Behavior Prior: Data-Efficient Human Models for Improved Human-AI Collaboration [0.5524804393257919]
  We show that using optimal behavior as a prior for human models makes these models vastly more data-efficient. We also show that using these improved human models often leads to better human-AI collaboration performance.
  arXiv  Detail & Related papers  (2022-11-03T06:10:22Z)
• It Takes Two: Learning to Plan for Human-Robot Cooperative Carrying [0.6981715773998527]
  We present a method for predicting realistic motion plans for cooperative human-robot teams on a table-carrying task. We use a Variational Recurrent Neural Network, VRNN, to model the variation in the trajectory of a human-robot team over time. We show that the model generates more human-like motion compared to a baseline, centralized sampling-based planner.
  arXiv  Detail & Related papers  (2022-09-26T17:59:23Z)
• Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination [21.800115245671737]
  We consider the problem of training a Reinforcement Learning (RL) agent without using any human data to make it capable of collaborating with humans. We derive a centralized population entropy objective to facilitate learning of a diverse population of agents, which is later used to train a robust agent to collaborate with unseen partners.
  arXiv  Detail & Related papers  (2021-12-22T07:19:36Z)
• Collaborating with Humans without Human Data [6.158826414652401]
  We study the problem of how to train agents that collaborate well with human partners without using human data. We train our agent partner as the best response to a population of self-play agents and their past checkpoints. We find that Fictitious Co-Play (FCP) agents score significantly higher than SP, PP, and BCP when paired with novel agent and human partners.
  arXiv  Detail & Related papers  (2021-10-15T16:03:57Z)
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback [82.96694147237113]
  We present Skill Preferences, an algorithm that learns a model over human preferences and uses it to extract human-aligned skills from offline data. We show that SkiP enables a simulated kitchen robot to solve complex multi-step manipulation tasks.
  arXiv  Detail & Related papers  (2021-08-11T18:04:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.