CORA: Coalitional Rational Advantage Decomposition for Multi-Agent Policy Gradients

• URL: http://arxiv.org/abs/2506.04265v2
• Date: Wed, 18 Jun 2025 11:50:18 GMT
• Title: CORA: Coalitional Rational Advantage Decomposition for Multi-Agent Policy Gradients
• Authors: Mengda Ji, Genjiu Xu, Liying Wang,
• Abstract summary: This work focuses on the credit assignment problem in cooperative multi-agent reinforcement learning (MARL)<n>We propose a credit assignment method called Coalitional Rational Advantage Decomposition (CORA)<n>Experiments on matrix games, differential games, and multi-agent collaboration benchmarks demonstrate that CORA outperforms strong baselines.
• Score: 1.1592964544667297
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work focuses on the credit assignment problem in cooperative multi-agent reinforcement learning (MARL). Sharing the global advantage among agents often leads to suboptimal policy updates as it fails to account for the distinct contributions of agents. Although numerous methods consider global or individual contributions for credit assignment, a detailed analysis at the coalition level remains lacking in many approaches. This work analyzes the over-updating problem during multi-agent policy updates from a coalition-level perspective. To address this issue, we propose a credit assignment method called Coalitional Rational Advantage Decomposition (CORA). CORA evaluates coalitional advantages via marginal contributions from all possible coalitions and decomposes advantages using the core solution from cooperative game theory, ensuring coalitional rationality. To reduce computational overhead, CORA employs random coalition sampling. Experiments on matrix games, differential games, and multi-agent collaboration benchmarks demonstrate that CORA outperforms strong baselines, particularly in tasks with multiple local optima. These findings highlight the importance of coalition-aware credit assignment for improving MARL performance.

Related papers

• Nucleolus Credit Assignment for Effective Coalitions in Multi-agent Reinforcement Learning [13.220552085613292]
  In cooperative multi-agent reinforcement learning (MARL), agents typically form a single grand coalition based on credit assignment to tackle a composite task.<n>This paper proposed a nucleolus-based credit assignment grounded in cooperative game theory, enabling the autonomous partitioning of agents into small coalitions.
  arXiv  Detail & Related papers  (2025-03-01T07:01:58Z)
• Reducing Optimism Bias in Incomplete Cooperative Games [0.0]
  We present a framework aimed at optimizing the sequence for revealing coalition values in cooperative games.<n>Our contributions are threefold: (i) we study the individual players' optimistic completions of games with missing coalition values along with the arising gap, and investigate its analytical characteristics that facilitate more efficient optimization; (ii) we develop methods to minimize this gap over classes of games with a known prior by disclosing values of additional coalitions in both offline and online fashion; and (iii) we empirically demonstrate the algorithms' performance in practical scenarios.
  arXiv  Detail & Related papers  (2024-02-02T21:58:26Z)
• Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning [48.41925886860991]
  Real-world cooperation often requires intensive coordination among agents simultaneously. Traditional methods that learn the value function as a monotonic mixing of per-agent utilities cannot solve the tasks with non-monotonic returns. We propose a novel explicit credit assignment method to address the non-monotonic problem.
  arXiv  Detail & Related papers  (2023-02-14T07:23:59Z)
• RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning [55.55009081609396]
  We propose a novel method, called Relation-Aware Credit Assignment (RACA), which achieves zero-shot generalization in ad-hoc cooperation scenarios. RACA takes advantage of a graph-based encoder relation to encode the topological structure between agents. Our method outperforms baseline methods on the StarCraftII micromanagement benchmark and ad-hoc cooperation scenarios.
  arXiv  Detail & Related papers  (2022-06-02T03:39:27Z)
• Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming [0.0]
  We show that reformulating an agent's policy to be conditional on the policies of its teammates inherently maximizes Mutual Information (MI) lower-bound when optimizing under Policy Gradient (PG) Our approach, InfoPG, outperforms baselines in learning emergent collaborative behaviors and sets the state-of-the-art in decentralized cooperative MARL tasks.
  arXiv  Detail & Related papers  (2022-01-20T22:54:32Z)
• Learning Cooperative Multi-Agent Policies with Partial Reward Decoupling [13.915157044948364]
  One of the preeminent obstacles to scaling multi-agent reinforcement learning is assigning credit to individual agents' actions. In this paper, we address this credit assignment problem with an approach that we call textitpartial reward decoupling (PRD) PRD decomposes large cooperative multi-agent RL problems into decoupled subproblems involving subsets of agents, thereby simplifying credit assignment.
  arXiv  Detail & Related papers  (2021-12-23T17:48:04Z)
• Locality Matters: A Scalable Value Decomposition Approach for Cooperative Multi-Agent Reinforcement Learning [52.7873574425376]
  Cooperative multi-agent reinforcement learning (MARL) faces significant scalability issues due to state and action spaces that are exponentially large in the number of agents. We propose a novel, value-based multi-agent algorithm called LOMAQ, which incorporates local rewards in the Training Decentralized Execution paradigm.
  arXiv  Detail & Related papers  (2021-09-22T10:08:15Z)
• Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning [34.856522993714535]
  We propose Shapley Counterfactual Credit Assignment, a novel method for explicit credit assignment which accounts for the coalition of agents. Our method outperforms existing cooperative MARL algorithms significantly and achieves the state-of-the-art, with especially large margins on tasks with more severe difficulties.
  arXiv  Detail & Related papers  (2021-06-01T07:38:34Z)
• Multi-agent Policy Optimization with Approximatively Synchronous Advantage Estimation [55.96893934962757]
  In multi-agent system, polices of different agents need to be evaluated jointly. In current methods, value functions or advantage functions use counter-factual joint actions which are evaluated asynchronously. In this work, we propose the approximatively synchronous advantage estimation.
  arXiv  Detail & Related papers  (2020-12-07T07:29:19Z)
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning [53.73686229912562]
  We propose a novel MARL approach called Universal Value Exploration (UneVEn) UneVEn learns a set of related tasks simultaneously with a linear decomposition of universal successor features. Empirical results on a set of exploration games, challenging cooperative predator-prey tasks requiring significant coordination among agents, and StarCraft II micromanagement benchmarks show that UneVEn can solve tasks where other state-of-the-art MARL methods fail.
  arXiv  Detail & Related papers  (2020-10-06T19:08:47Z)
• FACMAC: Factored Multi-Agent Centralised Policy Gradients [103.30380537282517]
  We propose FACtored Multi-Agent Centralised policy gradients (FACMAC) It is a new method for cooperative multi-agent reinforcement learning in both discrete and continuous action spaces. We evaluate FACMAC on variants of the multi-agent particle environments, a novel multi-agent MuJoCo benchmark, and a challenging set of StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2020-03-14T21:29:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.