Modelling Behavioural Diversity for Learning in Open-Ended Games

• URL: http://arxiv.org/abs/2103.07927v1
• Date: Sun, 14 Mar 2021 13:42:39 GMT
• Title: Modelling Behavioural Diversity for Learning in Open-Ended Games
• Authors: Nicolas Perez Nieves, Yaodong Yang, Oliver Slumbers, David Henry Mguni, Jun Wang
• Abstract summary: We offer a geometric interpretation of behavioural diversity in games. We introduce a novel diversity metric based on emphdeterminantal point processes (DPP) We prove the uniqueness of the diverse best response and the convergence of our algorithms on two-player games.
• Score: 15.978932309579013
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Promoting behavioural diversity is critical for solving games with non-transitive dynamics where strategic cycles exist, and there is no consistent winner (e.g., Rock-Paper-Scissors). Yet, there is a lack of rigorous treatment for defining diversity and constructing diversity-aware learning dynamics. In this work, we offer a geometric interpretation of behavioural diversity in games and introduce a novel diversity metric based on \emph{determinantal point processes} (DPP). By incorporating the diversity metric into best-response dynamics, we develop \emph{diverse fictitious play} and \emph{diverse policy-space response oracle} for solving normal-form games and open-ended games. We prove the uniqueness of the diverse best response and the convergence of our algorithms on two-player games. Importantly, we show that maximising the DPP-based diversity metric guarantees to enlarge the \emph{gamescape} -- convex polytopes spanned by agents' mixtures of strategies. To validate our diversity-aware solvers, we test on tens of games that show strong non-transitivity. Results suggest that our methods achieve much lower exploitability than state-of-the-art solvers by finding effective and diverse strategies.

Related papers

• Measuring Diversity of Game Scenarios [15.100151112002235]
  We aim to bridge the current gaps in literature and practice, offering insights into effective strategies for measuring and integrating diversity in game scenarios. This survey not only charts a path for future research in diverse game scenarios but also serves as a handbook for industry practitioners seeking to leverage diversity as a key component of game design and development.
  arXiv  Detail & Related papers  (2024-04-15T07:59:52Z)
• Diversify Question Generation with Retrieval-Augmented Style Transfer [68.00794669873196]
  We propose RAST, a framework for Retrieval-Augmented Style Transfer. The objective is to utilize the style of diverse templates for question generation. We develop a novel Reinforcement Learning (RL) based approach that maximizes a weighted combination of diversity reward and consistency reward.
  arXiv  Detail & Related papers  (2023-10-23T02:27:31Z)
• Learning Diverse Risk Preferences in Population-based Self-play [23.07952140353786]
  Current self-play algorithms optimize the agent to maximize expected win-rates against its current or historical copies. We introduce diversity from the perspective that agents could have diverse risk preferences in the face of uncertainty. We show that our method achieves comparable or superior performance in competitive games.
  arXiv  Detail & Related papers  (2023-05-19T06:56:02Z)
• On the Convergence of No-Regret Learning Dynamics in Time-Varying Games [89.96815099996132]
  We characterize the convergence of optimistic gradient descent (OGD) in time-varying games. Our framework yields sharp convergence bounds for the equilibrium gap of OGD in zero-sum games. We also provide new insights on dynamic regret guarantees in static games.
  arXiv  Detail & Related papers  (2023-01-26T17:25:45Z)
• Finding mixed-strategy equilibria of continuous-action games without gradients using randomized policy networks [83.28949556413717]
  We study the problem of computing an approximate Nash equilibrium of continuous-action game without access to gradients. We model players' strategies using artificial neural networks. This paper is the first to solve general continuous-action games with unrestricted mixed strategies and without any gradient information.
  arXiv  Detail & Related papers  (2022-11-29T05:16:41Z)
• Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity [49.68758494467258]
  We study how to construct diverse populations of agents by carefully structuring how individuals within a population interact. Our approach is based on interaction graphs, which control the flow of information between agents during training. We provide evidence for the importance of diversity in multi-agent training and analyse the effect of applying different interaction graphs on the training trajectories, diversity and performance of populations in a range of games.
  arXiv  Detail & Related papers  (2021-10-08T11:29:52Z)
• Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games [44.30509625560908]
  In open-ended learning algorithms, there are no widely accepted definitions for diversity, making it hard to construct and evaluate the diverse policies. We propose a unified measure of diversity in multi-agent open-ended learning based on both Behavioral Diversity (BD) and Response Diversity (RD) We show that many current diversity measures fall in one of the categories of BD or RD but not both. With this unified diversity measure, we design the corresponding diversity-promoting objective and population effectivity when seeking the best responses in open-ended learning.
  arXiv  Detail & Related papers  (2021-06-09T10:11:06Z)
• Generating Diverse and Competitive Play-Styles for Strategy Games [58.896302717975445]
  We propose Portfolio Monte Carlo Tree Search with Progressive Unpruning for playing a turn-based strategy game (Tribes) We show how it can be parameterized so a quality-diversity algorithm (MAP-Elites) is used to achieve different play-styles while keeping a competitive level of play. Our results show that this algorithm is capable of achieving these goals even for an extensive collection of game levels beyond those used for training.
  arXiv  Detail & Related papers  (2021-04-17T20:33:24Z)
• Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization [42.33734089361143]
  We propose a technique for discovering diverse strategic policies in complex multi-agent games. We derive a new algorithm, Reward-Randomized Policy Gradient (RPG) RPG is able to discover multiple distinctive human-interpretable strategies in challenging temporal trust dilemmas.
  arXiv  Detail & Related papers  (2021-03-08T06:26:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.