Counterfactual Conditional Likelihood Rewards for Multiagent Exploration

• URL: http://arxiv.org/abs/2602.11740v1
• Date: Thu, 12 Feb 2026 09:08:49 GMT
• Title: Counterfactual Conditional Likelihood Rewards for Multiagent Exploration
• Abstract summary: We introduce Counterfactual Conditional Likelihood rewards, which score each agent's exploration by isolating its unique contribution to team exploration.<n>Experiments in continuous multiagent domains show that CCL rewards accelerate learning for domains with sparse team rewards, where most joint actions yield zero rewards.
• Score: 1.0705399532413618
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Efficient exploration is critical for multiagent systems to discover coordinated strategies, particularly in open-ended domains such as search and rescue or planetary surveying. However, when exploration is encouraged only at the individual agent level, it often leads to redundancy, as agents act without awareness of how their teammates are exploring. In this work, we introduce Counterfactual Conditional Likelihood (CCL) rewards, which score each agent's exploration by isolating its unique contribution to team exploration. Unlike prior methods that reward agents solely for the novelty of their individual observations, CCL emphasizes observations that are informative with respect to the joint exploration of the team. Experiments in continuous multiagent domains show that CCL rewards accelerate learning for domains with sparse team rewards, where most joint actions yield zero rewards, and are particularly effective in tasks that require tight coordination among agents.

Related papers

• CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery [67.47220507982522]
  Large language model (LLM)-based evolution is a promising approach for openended discovery.<n>We present CORAL, the first framework for autonomous multi-agent evolution on open-ended problems.<n>CORAL replaces rigid control with long-running agents that explore, reflect, and collaborate.
  arXiv  Detail & Related papers  (2026-04-02T05:59:06Z)
• Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning [9.448662880144845]
  Individual Contributions as intrinsic Exploration Scaffolds (ICES) ICES is a novel approach to motivate exploration by assessing each agent's contribution from a global view. Experiments on cooperative benchmark tasks with sparse rewards, including Google Research Football (GRF) and StarCraft Multi-agent Challenge (SMAC)
  arXiv  Detail & Related papers  (2024-05-28T12:18:19Z)
• Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing [34.299478481229265]
  We propose MACE, a simple yet effective multi-agent coordinated exploration method. By communicating only local novelty, agents can take into account other agents' local novelty to approximate the global novelty. We show that MACE achieves superior performance in three multi-agent environments with sparse rewards.
  arXiv  Detail & Related papers  (2024-02-03T09:35:25Z)
• Curiosity-Driven Multi-Agent Exploration with Mixed Objectives [7.247148291603988]
  Intrinsic rewards have been increasingly used to mitigate the sparse reward problem in single-agent reinforcement learning. Curiosity-driven exploration is a simple yet efficient approach that quantifies this novelty as the prediction error of the agent's curiosity module. We show here, however, that naively using this curiosity-driven approach to guide exploration in sparse reward cooperative multi-agent environments does not consistently lead to improved results.
  arXiv  Detail & Related papers  (2022-10-29T02:45:38Z)
• Multi-agent Deep Covering Skill Discovery [50.812414209206054]
  We propose Multi-agent Deep Covering Option Discovery, which constructs the multi-agent options through minimizing the expected cover time of the multiple agents' joint state space. Also, we propose a novel framework to adopt the multi-agent options in the MARL process. We show that the proposed algorithm can effectively capture the agent interactions with the attention mechanism, successfully identify multi-agent options, and significantly outperforms prior works using single-agent options or no options.
  arXiv  Detail & Related papers  (2022-10-07T00:40:59Z)
• Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration [40.87053312548429]
  We introduce a novel Episodic Multi-agent reinforcement learning with Curiosity-driven exploration, called EMC. We use prediction errors of individual Q-values as intrinsic rewards for coordinated exploration and utilize episodic memory to exploit explored informative experience to boost policy training.
  arXiv  Detail & Related papers  (2021-11-22T07:34:47Z)
• Cooperative Exploration for Multi-Agent Deep Reinforcement Learning [127.4746863307944]
  We propose cooperative multi-agent exploration (CMAE) for deep reinforcement learning. The goal is selected from multiple projected state spaces via a normalized entropy-based technique. We demonstrate that CMAE consistently outperforms baselines on various tasks.
  arXiv  Detail & Related papers  (2021-07-23T20:06:32Z)
• Explore and Control with Adversarial Surprise [78.41972292110967]
  Reinforcement learning (RL) provides a framework for learning goal-directed policies given user-specified rewards. We propose a new unsupervised RL technique based on an adversarial game which pits two policies against each other to compete over the amount of surprise an RL agent experiences. We show that our method leads to the emergence of complex skills by exhibiting clear phase transitions.
  arXiv  Detail & Related papers  (2021-07-12T17:58:40Z)
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning [53.73686229912562]
  We propose a novel MARL approach called Universal Value Exploration (UneVEn) UneVEn learns a set of related tasks simultaneously with a linear decomposition of universal successor features. Empirical results on a set of exploration games, challenging cooperative predator-prey tasks requiring significant coordination among agents, and StarCraft II micromanagement benchmarks show that UneVEn can solve tasks where other state-of-the-art MARL methods fail.
  arXiv  Detail & Related papers  (2020-10-06T19:08:47Z)
• Learning to Incentivize Other Learning Agents [73.03133692589532]
  We show how to equip RL agents with the ability to give rewards directly to other agents, using a learned incentive function. Such agents significantly outperform standard RL and opponent-shaping agents in challenging general-sum Markov games. Our work points toward more opportunities and challenges along the path to ensure the common good in a multi-agent future.
  arXiv  Detail & Related papers  (2020-06-10T20:12:38Z)
• Scalable Multi-Agent Inverse Reinforcement Learning via Actor-Attention-Critic [54.2180984002807]
  Multi-agent adversarial inverse reinforcement learning (MA-AIRL) is a recent approach that applies single-agent AIRL to multi-agent problems. We propose a multi-agent inverse RL algorithm that is more sample-efficient and scalable than previous works.
  arXiv  Detail & Related papers  (2020-02-24T20:30:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.