Learning to Play Soccer From Scratch: Sample-Efficient Emergent Coordination through Curriculum-Learning and Competition

• URL: http://arxiv.org/abs/2103.05174v1
• Date: Tue, 9 Mar 2021 01:57:16 GMT
• Title: Learning to Play Soccer From Scratch: Sample-Efficient Emergent Coordination through Curriculum-Learning and Competition
• Authors: Pavan Samtani, Francisco Leiva, Javier Ruiz-del-Solar
• Abstract summary: This work proposes a scheme that allows learning complex multi-agent behaviors in a sample efficient manner, applied to 2v2 soccer. The problem is formulated as a Markov game, and solved using deep reinforcement learning. Our results show that high quality soccer play can be obtained with our approach in just under 40M interactions.
• Score: 1.675857332621569
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work proposes a scheme that allows learning complex multi-agent behaviors in a sample efficient manner, applied to 2v2 soccer. The problem is formulated as a Markov game, and solved using deep reinforcement learning. We propose a basic multi-agent extension of TD3 for learning the policy of each player, in a decentralized manner. To ease learning, the task of 2v2 soccer is divided in three stages: 1v0, 1v1 and 2v2. The process of learning in multi-agent stages (1v1 and 2v2) uses agents trained on a previous stage as fixed opponents. In addition, we propose using experience sharing, a method that shares experience from a fixed opponent, trained in a previous stage, for training the agent currently learning, and a form of frame-skipping, to raise performance significantly. Our results show that high quality soccer play can be obtained with our approach in just under 40M interactions. A summarized video of the resulting game play can be found in https://youtu.be/f25l1j1U9RM.

Related papers

• Enhancing Two-Player Performance Through Single-Player Knowledge Transfer: An Empirical Study on Atari 2600 Games [1.03590082373586]
  This study examines the proposed idea in ten different Atari 2600 environments using the Atari 2600 RAM as the input state. We discuss the advantages of using transfer learning from a single-player training process over training in a two-player setting from scratch.
  arXiv  Detail & Related papers  (2024-10-22T02:57:44Z)
• Leading the Pack: N-player Opponent Shaping [52.682734939786464]
  We extend Opponent Shaping (OS) methods to environments involving multiple co-players and multiple shaping agents. We find that when playing with a large number of co-players, OS methods' relative performance reduces, suggesting that in the limit OS methods may not perform well.
  arXiv  Detail & Related papers  (2023-12-19T20:01:42Z)
• All by Myself: Learning Individualized Competitive Behaviour with a Contrastive Reinforcement Learning optimization [57.615269148301515]
  In a competitive game scenario, a set of agents have to learn decisions that maximize their goals and minimize their adversaries' goals at the same time. We propose a novel model composed of three neural layers that learn a representation of a competitive game, learn how to map the strategy of specific opponents, and how to disrupt them. Our experiments demonstrate that our model achieves better performance when playing against offline, online, and competitive-specific models, in particular when playing against the same opponent multiple times.
  arXiv  Detail & Related papers  (2023-10-02T08:11:07Z)
• Value-based CTDE Methods in Symmetric Two-team Markov Game: from Cooperation to Team Competition [3.828689444527739]
  We evaluate cooperative value-based methods in a mixed cooperative-competitive environment. We selected three training methods based on the centralised training and decentralised execution paradigm. For our experiments, we modified the StarCraft Multi-Agent Challenge environment to create competitive environments where both teams could learn and compete simultaneously.
  arXiv  Detail & Related papers  (2022-11-21T22:25:55Z)
• Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning [95.78031053296513]
  No-press Diplomacy is a complex strategy game involving both cooperation and competition. We introduce a planning algorithm we call DiL-piKL that regularizes a reward-maximizing policy toward a human imitation-learned policy. We show that DiL-piKL can be extended into a self-play reinforcement learning algorithm we call RL-DiL-piKL.
  arXiv  Detail & Related papers  (2022-10-11T14:47:35Z)
• TiKick: Toward Playing Multi-agent Football Full Games from Single-agent Demonstrations [31.596018856092513]
  Tikick is the first learning-based AI system that can take over the multi-agent Google Research Football full game. To the best of our knowledge, Tikick is the first learning-based AI system that can take over the multi-agent Google Research Football full game.
  arXiv  Detail & Related papers  (2021-10-09T08:34:58Z)
• Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition [88.26752130107259]
  In real-world multiagent systems, agents with different capabilities may join or leave without altering the team's overarching goals. We propose COPA, a coach-player framework to tackle this problem. We 1) adopt the attention mechanism for both the coach and the players; 2) propose a variational objective to regularize learning; and 3) design an adaptive communication method to let the coach decide when to communicate with the players.
  arXiv  Detail & Related papers  (2021-05-18T17:27:37Z)
• Two-stage training algorithm for AI robot soccer [2.0757564643017092]
  Two-stage heterogeneous centralized training is proposed to improve the learning performance of heterogeneous agents. The proposed method is applied to 5 versus 5 AI robot soccer for validation.
  arXiv  Detail & Related papers  (2021-04-13T04:24:13Z)
• An analysis of Reinforcement Learning applied to Coach task in IEEE Very Small Size Soccer [2.5400028272658144]
  This paper proposes an end-to-end approach for the coaching task based on Reinforcement Learning (RL) We trained two RL policies against three different teams in a simulated environment. Our results were assessed against one of the top teams of the VSSS league.
  arXiv  Detail & Related papers  (2020-11-23T23:10:06Z)
• Multi-Agent Collaboration via Reward Attribution Decomposition [75.36911959491228]
  We propose Collaborative Q-learning (CollaQ) that achieves state-of-the-art performance in the StarCraft multi-agent challenge. CollaQ is evaluated on various StarCraft Attribution maps and shows that it outperforms existing state-of-the-art techniques.
  arXiv  Detail & Related papers  (2020-10-16T17:42:11Z)
• Learning from Learners: Adapting Reinforcement Learning Agents to be Competitive in a Card Game [71.24825724518847]
  We present a study on how popular reinforcement learning algorithms can be adapted to learn and to play a real-world implementation of a competitive multiplayer card game. We propose specific training and validation routines for the learning agents, in order to evaluate how the agents learn to be competitive and explain how they adapt to each others' playing style.
  arXiv  Detail & Related papers  (2020-04-08T14:11:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.