Learning to Advise and Learning from Advice in Cooperative Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2205.11163v1
• Date: Mon, 23 May 2022 09:56:12 GMT
• Title: Learning to Advise and Learning from Advice in Cooperative Multi-Agent Reinforcement Learning
• Abstract summary: Learning toAdvise and Learning from Advice (LALA) is proposed to improve multi-agent reinforcement learning (MARL) The advisor learns to aggregate decision information in both spatial and temporal domains and coordinated decisions by employing a dualtemporal convolutional neural network with a task-oriented objective function. Experimental results indicate the advantage of LALA over baseline approaches in terms of both learning efficiency and coordination capability.
• Score: 10.691232629333413
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to coordinate is a daunting problem in multi-agent reinforcement learning (MARL). Previous works have explored it from many facets, including cognition between agents, credit assignment, communication, expert demonstration, etc. However, less attention were paid to agents' decision structure and the hierarchy of coordination. In this paper, we explore the spatiotemporal structure of agents' decisions and consider the hierarchy of coordination from the perspective of multilevel emergence dynamics, based on which a novel approach, Learning to Advise and Learning from Advice (LALA), is proposed to improve MARL. Specifically, by distinguishing the hierarchy of coordination, we propose to enhance decision coordination at meso level with an advisor and leverage a policy discriminator to advise agents' learning at micro level. The advisor learns to aggregate decision information in both spatial and temporal domains and generates coordinated decisions by employing a spatiotemporal dual graph convolutional neural network with a task-oriented objective function. Each agent learns from the advice via a policy generative adversarial learning method where a discriminator distinguishes between the policies of the agent and the advisor and boosts both of them based on its judgement. Experimental results indicate the advantage of LALA over baseline approaches in terms of both learning efficiency and coordination capability. Coordination mechanism is investigated from the perspective of multilevel emergence dynamics and mutual information point of view, which provides a novel perspective and method to analyze and improve MARL algorithms.

Related papers

• Structured LLM Reasoning for Zero-Shot Human--Robot Coordination Under Hidden Goals [3.158926173252809]
  We present a structured large-language-model (LLM) architecture for zero-shot human--robot coordination in a cooperative construction task with private goal views.<n> Guided by a Dec-POMDP formulation, the architecture decomposes decision-making into (i) action-conditioned Theory-of-Mind (ToM) inference, (ii) hierarchical planning, (iii) conversation interpretation, (iv) action verification, and (v) feedback-based replanning.
  arXiv  Detail & Related papers  (2026-08-05T00:40:03Z)
• Bridging Talk and Thought: Understanding Dialogue Dynamics Across Collaborative Problem-Solving Contexts [57.79777631004424]
  We present a conceptual framework for analyzing dialogue in collaborative problem-solving contexts.<n>Our framework integrates cognitive and non-cognitive problem solving with metacognitive regulatory mechanisms.
  arXiv  Detail & Related papers  (2026-06-25T16:20:49Z)
• Agentic Reasoning for Large Language Models [122.81018455095999]
  Reasoning is a fundamental cognitive process underlying inference, problem-solving, and decision-making.<n>Large language models (LLMs) demonstrate strong reasoning capabilities in closed-world settings, but struggle in open-ended and dynamic environments.<n>Agentic reasoning marks a paradigm shift by reframing LLMs as autonomous agents that plan, act, and learn through continual interaction.
  arXiv  Detail & Related papers  (2026-01-18T18:58:23Z)
• Joint Optimization of Cooperation Efficiency and Communication Covertness for Target Detection with AUVs [105.81167650318054]
  This paper investigates underwater cooperative target detection using autonomous underwater vehicles (AUVs)<n>We first formulate a joint trajectory and power control optimization problem, and then present an innovative hierarchical action management framework to solve it.<n>Under the centralized training and decentralized execution paradigm, our target detection framework enables adaptive covert cooperation while satisfying both energy and mobility constraints.
  arXiv  Detail & Related papers  (2025-10-21T02:14:11Z)
• Strategic Coordination for Evolving Multi-agent Systems: A Hierarchical Reinforcement and Collective Learning Approach [0.0]
  Reinforcement learning offers a way to model sequential decision-making.<n>Agents take high-level strategies using MARL to group possible plans for action space reduction.<n>Low-level collective learning layer ensures efficient and decentralized coordinated decisions.
  arXiv  Detail & Related papers  (2025-09-22T17:58:45Z)
• A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems [93.8285345915925]
  Reasoning is a fundamental cognitive process that enables logical inference, problem-solving, and decision-making. With the rapid advancement of large language models (LLMs), reasoning has emerged as a key capability that distinguishes advanced AI systems. We categorize existing methods along two dimensions: (1) Regimes, which define the stage at which reasoning is achieved; and (2) Architectures, which determine the components involved in the reasoning process.
  arXiv  Detail & Related papers  (2025-04-12T01:27:49Z)
• Contextual Knowledge Sharing in Multi-Agent Reinforcement Learning with Decentralized Communication and Coordination [0.9776703963093367]
  Multi-Agent Reinforcement Learning (Dec-MARL) has emerged as a pivotal approach for addressing complex tasks in dynamic environments. This paper presents a novel Dec-MARL framework that integrates peer-to-peer communication and coordination, incorporating goal-awareness and time-awareness into the agents' knowledge-sharing processes.
  arXiv  Detail & Related papers  (2025-01-26T22:49:50Z)
• Tacit Learning with Adaptive Information Selection for Cooperative Multi-Agent Reinforcement Learning [13.918498667158119]
  We introduce a novel cooperative MARL framework based on information selection and tacit learning. We integrate gating and selection mechanisms, allowing agents to adaptively filter information based on environmental changes. Experiments on popular MARL benchmarks show that our framework can be seamlessly integrated with state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-12-20T07:55:59Z)
• Multi-agent cooperation through learning-aware policy gradients [53.63948041506278]
  Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning. We present the first unbiased, higher-derivative-free policy gradient algorithm for learning-aware reinforcement learning. We derive from the iterated prisoner's dilemma a novel explanation for how and when cooperation arises among self-interested learning-aware agents.
  arXiv  Detail & Related papers  (2024-10-24T10:48:42Z)
• Hierarchical Consensus-Based Multi-Agent Reinforcement Learning for Multi-Robot Cooperation Tasks [17.914928652949314]
  We introduce the Hierarchical Consensus-based Multi-Agent Reinforcement Learning (HC-MARL) framework to address this limitation. HC-MARL employs contrastive learning to foster a global consensus among agents, enabling cooperative behavior without direct communication. To cater to the dynamic requirements of various tasks, consensus is divided into multiple layers, encompassing both short-term and long-term considerations.
  arXiv  Detail & Related papers  (2024-07-11T03:55:55Z)
• Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning [51.52387511006586]
  We propose Hierarchical Opponent modeling and Planning (HOP), a novel multi-agent decision-making algorithm. HOP is hierarchically composed of two modules: an opponent modeling module that infers others' goals and learns corresponding goal-conditioned policies. HOP exhibits superior few-shot adaptation capabilities when interacting with various unseen agents, and excels in self-play scenarios.
  arXiv  Detail & Related papers  (2024-06-12T08:48:06Z)
• LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models [75.89014602596673]
  Strategic reasoning requires understanding and predicting adversary actions in multi-agent settings while adjusting strategies accordingly. We explore the scopes, applications, methodologies, and evaluation metrics related to strategic reasoning with Large Language Models. It underscores the importance of strategic reasoning as a critical cognitive capability and offers insights into future research directions and potential improvements.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• Learning Reward Machines in Cooperative Multi-Agent Tasks [75.79805204646428]
  This paper presents a novel approach to Multi-Agent Reinforcement Learning (MARL) It combines cooperative task decomposition with the learning of reward machines (RMs) encoding the structure of the sub-tasks. The proposed method helps deal with the non-Markovian nature of the rewards in partially observable environments.
  arXiv  Detail & Related papers  (2023-03-24T15:12:28Z)
• A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning [17.893310647034188]
  We propose a new mutual information framework for multi-agent reinforcement learning. Applying policy to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic.
  arXiv  Detail & Related papers  (2023-03-01T12:21:30Z)
• RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning [55.55009081609396]
  We propose a novel method, called Relation-Aware Credit Assignment (RACA), which achieves zero-shot generalization in ad-hoc cooperation scenarios. RACA takes advantage of a graph-based encoder relation to encode the topological structure between agents. Our method outperforms baseline methods on the StarCraftII micromanagement benchmark and ad-hoc cooperation scenarios.
  arXiv  Detail & Related papers  (2022-06-02T03:39:27Z)
• Curriculum-Driven Multi-Agent Learning and the Role of Implicit Communication in Teamwork [24.92668968807012]
  We propose a curriculum-driven learning strategy for solving difficult multi-agent coordination tasks. We argue that emergent implicit communication plays a large role in enabling superior levels of coordination.
  arXiv  Detail & Related papers  (2021-06-21T14:54:07Z)
• Context-Aware Sparse Deep Coordination Graphs [20.582393720212547]
  Learning sparse coordination graphs adaptive to the coordination dynamics among agents is a long-standing problem in cooperative multi-agent learning. This paper proposes several value-based and observation-based schemes for learning dynamic topologies and evaluating them on a new Multi-Agent COordination (MACO) benchmark. By analyzing the individual advantages of each learning scheme on each type of problem and their overall performance, we propose a novel method using the variance of utility difference functions to learn context-aware sparse coordination topologies.
  arXiv  Detail & Related papers  (2021-06-05T12:59:03Z)
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning [53.73686229912562]
  We propose a novel MARL approach called Universal Value Exploration (UneVEn) UneVEn learns a set of related tasks simultaneously with a linear decomposition of universal successor features. Empirical results on a set of exploration games, challenging cooperative predator-prey tasks requiring significant coordination among agents, and StarCraft II micromanagement benchmarks show that UneVEn can solve tasks where other state-of-the-art MARL methods fail.
  arXiv  Detail & Related papers  (2020-10-06T19:08:47Z)
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks [11.480994804659908]
  Multi-agent deep reinforcement learning (MARL) suffers from a lack of commonly-used evaluation tasks and criteria. We provide a systematic evaluation and comparison of three different classes of MARL algorithms. Our experiments serve as a reference for the expected performance of algorithms across different learning tasks.
  arXiv  Detail & Related papers  (2020-06-14T11:22:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.