Learning Independently from Causality in Multi-Agent Environments

• URL: http://arxiv.org/abs/2311.02741v1
• Date: Sun, 5 Nov 2023 19:12:08 GMT
• Title: Learning Independently from Causality in Multi-Agent Environments
• Authors: Rafael Pina, Varuna De Silva and Corentin Artaud
• Abstract summary: Multi-Agent Reinforcement Learning (MARL) comprises an area of growing interest in the field of machine learning. The lazy agent pathology is a famous problem in MARL that denotes the event when some of the agents in a MARL team do not contribute to the common goal. We study a fully decentralised MARL setup where agents need to learn cooperation strategies and show that there is a causal relation between individual observations and the team reward.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-Agent Reinforcement Learning (MARL) comprises an area of growing interest in the field of machine learning. Despite notable advances, there are still problems that require investigation. The lazy agent pathology is a famous problem in MARL that denotes the event when some of the agents in a MARL team do not contribute to the common goal, letting the teammates do all the work. In this work, we aim to investigate this problem from a causality-based perspective. We intend to create the bridge between the fields of MARL and causality and argue about the usefulness of this link. We study a fully decentralised MARL setup where agents need to learn cooperation strategies and show that there is a causal relation between individual observations and the team reward. The experiments carried show how this relation can be used to improve independent agents in MARL, resulting not only on better performances as a team but also on the rise of more intelligent behaviours on individual agents.

Related papers

• The Heterogeneous Multi-Agent Challenge [0.4199844472131922]
  Multi-Agent Reinforcement Learning (MARL) is a growing research area which gained significant traction in recent years.<n> Heterogeneous Multi-Agent Reinforcement Learning (HeMARL) is where agents with different sensors, resources, or capabilities must cooperate based on local information.
  arXiv  Detail & Related papers  (2025-09-23T19:30:30Z)
• Corrupted by Reasoning: Reasoning Language Models Become Free-Riders in Public Goods Games [87.5673042805229]
  How large language models balance self-interest and collective well-being is a critical challenge for ensuring alignment, robustness, and safe deployment.<n>We adapt a public goods game with institutional choice from behavioral economics, allowing us to observe how different LLMs navigate social dilemmas.<n>Surprisingly, we find that reasoning LLMs, such as the o1 series, struggle significantly with cooperation.
  arXiv  Detail & Related papers  (2025-06-29T15:02:47Z)
• A Survey on the Memory Mechanism of Large Language Model based Agents [66.4963345269611]
  Large language model (LLM) based agents have recently attracted much attention from the research and industry communities. LLM-based agents are featured in their self-evolving capability, which is the basis for solving real-world problems. The key component to support agent-environment interactions is the memory of the agents.
  arXiv  Detail & Related papers  (2024-04-21T01:49:46Z)
• Fully Independent Communication in Multi-Agent Reinforcement Learning [4.470370168359807]
  Multi-Agent Reinforcement Learning (MARL) comprises a broad area of research within the field of multi-agent systems. We investigate how independent learners in MARL that do not share parameters can communicate. Our results show that, despite the challenges, independent agents can still learn communication strategies following our method.
  arXiv  Detail & Related papers  (2024-01-26T18:42:01Z)
• Discovering Causality for Efficient Cooperation in Multi-Agent Environments [0.0]
  In cooperative Multi-Agent Reinforcement Learning (MARL) agents are required to learn behaviours as a team to achieve a common goal. While learning a task, some agents may end up learning sub-optimal policies, not contributing to the objective of the team. Such agents are called lazy agents due to their non-cooperative behaviours that may arise from failing to understand whether they caused the rewards.
  arXiv  Detail & Related papers  (2023-06-20T18:56:25Z)
• Causality Detection for Efficient Multi-Agent Reinforcement Learning [0.0]
  We show how causality can be used to penalise lazy agents and improve their behaviours. We show empirically that using causality estimations in Multi-Agent Reinforcement Learning improves not only the holistic performance of the team, but also the individual capabilities of each agent.
  arXiv  Detail & Related papers  (2023-03-24T18:47:44Z)
• Learning Reward Machines in Cooperative Multi-Agent Tasks [75.79805204646428]
  This paper presents a novel approach to Multi-Agent Reinforcement Learning (MARL) It combines cooperative task decomposition with the learning of reward machines (RMs) encoding the structure of the sub-tasks. The proposed method helps deal with the non-Markovian nature of the rewards in partially observable environments.
  arXiv  Detail & Related papers  (2023-03-24T15:12:28Z)
• On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning [55.95253619768565]
  Current MARL algorithms assume that the number of agents within a group remains fixed throughout an experiment. In many practical problems, an agent may terminate before their teammates. We present a novel architecture for an existing state-of-the-art MARL algorithm which uses attention instead of a fully connected layer with absorbing states.
  arXiv  Detail & Related papers  (2021-11-10T23:45:08Z)
• LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates [33.28389165779892]
  In cooperative multi-agent reinforcement learning (MARL), where agents only have access to partial observations, efficiently leveraging local information is critical. We propose a novel framework, multi-agent textitLocal INformation Decomposition for Awareness of teammates (LINDA), with which agents learn to decompose local information and build awareness for each teammate.
  arXiv  Detail & Related papers  (2021-09-26T06:46:51Z)
• Joint Attention for Multi-Agent Coordination and Social Learning [108.31232213078597]
  We show that joint attention can be useful as a mechanism for improving multi-agent coordination and social learning. Joint attention leads to higher performance than a competitive centralized critic baseline across multiple environments. Taken together, these findings suggest that joint attention may be a useful inductive bias for multi-agent learning.
  arXiv  Detail & Related papers  (2021-04-15T20:14:19Z)
• Game-Theoretic Multiagent Reinforcement Learning [73.76806922670379]
  Multiagent reinforcement learning (MARL) corresponds to the learning problem in a multiagent system in which multiple agents learn simultaneously.<n>Despite great successes in MARL, there is a lack of a self-contained overview of the literature that covers game-theoretic foundations of modern MARL methods.<n>This work provides a self-contained assessment of the current state-of-the-art MARL techniques from a game-theoretic perspective.
  arXiv  Detail & Related papers  (2020-11-01T18:24:40Z)
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning [53.73686229912562]
  We propose a novel MARL approach called Universal Value Exploration (UneVEn) UneVEn learns a set of related tasks simultaneously with a linear decomposition of universal successor features. Empirical results on a set of exploration games, challenging cooperative predator-prey tasks requiring significant coordination among agents, and StarCraft II micromanagement benchmarks show that UneVEn can solve tasks where other state-of-the-art MARL methods fail.
  arXiv  Detail & Related papers  (2020-10-06T19:08:47Z)
• Learning to Incentivize Other Learning Agents [73.03133692589532]
  We show how to equip RL agents with the ability to give rewards directly to other agents, using a learned incentive function. Such agents significantly outperform standard RL and opponent-shaping agents in challenging general-sum Markov games. Our work points toward more opportunities and challenges along the path to ensure the common good in a multi-agent future.
  arXiv  Detail & Related papers  (2020-06-10T20:12:38Z)
• On Solving Cooperative MARL Problems with a Few Good Experiences [8.596915685049511]
  Cooperative Multi-agent Reinforcement Learning (MARL) is crucial for cooperative decentralized decision learning. In many domains such as search and rescue, drone surveillance, package delivery and fire fighting problems, a key challenge is learning with a few good experiences. We provide a novel fictitious self imitation approach that is able to simultaneously handle non-stationarity and sparse good experiences.
  arXiv  Detail & Related papers  (2020-01-22T12:53:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.