Accumulating Risk Capital Through Investing in Cooperation

• URL: http://arxiv.org/abs/2101.10305v1
• Date: Mon, 25 Jan 2021 18:41:45 GMT
• Title: Accumulating Risk Capital Through Investing in Cooperation
• Authors: Charlotte Roman, Michael Dennis, Andrew Critch, Stuart Russell
• Abstract summary: We show that the trade-off between safety and cooperation is not severe, and you can receive exponentially large returns through cooperation from a small amount of risk. We propose a method for training policies that targets this objective, Accumulating Risk Capital Through Investing in Cooperation (ARCTIC)
• Score: 12.053132866404972
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work on promoting cooperation in multi-agent learning has resulted in many methods which successfully promote cooperation at the cost of becoming more vulnerable to exploitation by malicious actors. We show that this is an unavoidable trade-off and propose an objective which balances these concerns, promoting both safety and long-term cooperation. Moreover, the trade-off between safety and cooperation is not severe, and you can receive exponentially large returns through cooperation from a small amount of risk. We study both an exact solution method and propose a method for training policies that targets this objective, Accumulating Risk Capital Through Investing in Cooperation (ARCTIC), and evaluate them in iterated Prisoner's Dilemma and Stag Hunt.

Related papers

• In Which Areas of Technical AI Safety Could Geopolitical Rivals Cooperate? [66.89036079974998]
  We consider technical factors that impact the risks of international cooperation on AI safety research. We focus on the degree to which such cooperation can advance dangerous capabilities, result in the sharing of sensitive information, or provide opportunities for harm. We argue that existing frameworks for managing associated risks can be supplemented with consideration of key risks specific to cooperation on technical AI safety research.
  arXiv  Detail & Related papers  (2025-04-17T13:03:56Z)
• Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systems [1.2564343689544843]
  We develop simulations of AI agents collaborating on shared objectives to study security risks and trade-offs. We observe infectious malicious prompts - the multi-hop spreading of malicious instructions. Our findings illustrate potential trade-off between security and collaborative efficiency in multi-agent systems.
  arXiv  Detail & Related papers  (2025-02-26T14:00:35Z)
• Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics [9.263837897126871]
  Multi-Agent Reinforcement Learning (MARL) has gained significant traction for solving complex real-world tasks. The inherentity and uncertainty in these environments pose substantial challenges to efficient and robust policy learning. We propose a novel approach that integrates distributional learning with a safety-focused loss function to improve convergence in cooperative MARL tasks.
  arXiv  Detail & Related papers  (2025-01-21T11:31:01Z)
• CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperation [98.11670473661587]
  CaPo improves cooperation efficiency with two phases: 1) meta-plan generation, and 2) progress-adaptive meta-plan and execution. Experimental results on the ThreeDworld Multi-Agent Transport and Communicative Watch-And-Help tasks demonstrate that CaPo achieves much higher task completion rate and efficiency compared with state-of-the-arts.
  arXiv  Detail & Related papers  (2024-11-07T13:08:04Z)
• Multi-agent cooperation through learning-aware policy gradients [53.63948041506278]
  Self-interested individuals often fail to cooperate, posing a fundamental challenge for multi-agent learning. We present the first unbiased, higher-derivative-free policy gradient algorithm for learning-aware reinforcement learning. We derive from the iterated prisoner's dilemma a novel explanation for how and when cooperation arises among self-interested learning-aware agents.
  arXiv  Detail & Related papers  (2024-10-24T10:48:42Z)
• Cognitive Insights and Stable Coalition Matching for Fostering Multi-Agent Cooperation [6.536780912510439]
  We propose a novel matching coalition mechanism that leverages the strengths of agents with different ToM levels. Our work demonstrates the potential of leveraging ToM to create more sophisticated and human-like coordination strategies.
  arXiv  Detail & Related papers  (2024-05-28T10:59:33Z)
• Emergent Cooperation under Uncertain Incentive Alignment [7.906156032228933]
  We study how cooperation can arise among reinforcement learning agents in scenarios characterised by infrequent encounters. We study the effects of mechanisms, such as reputation and intrinsic rewards, that have been proposed in the literature to foster cooperation in mixed-motives environments.
  arXiv  Detail & Related papers  (2024-01-23T10:55:54Z)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
  This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
  arXiv  Detail & Related papers  (2023-12-18T16:09:43Z)
• A Hierarchical Approach to Population Training for Human-AI Collaboration [20.860808795671343]
  We introduce a Hierarchical Reinforcement Learning (HRL) based method for Human-AI Collaboration. We demonstrate that our method is able to dynamically adapt to novel partners of different play styles and skill levels in the 2-player collaborative Overcooked game environment.
  arXiv  Detail & Related papers  (2023-05-26T07:53:12Z)
• Learning Reward Machines in Cooperative Multi-Agent Tasks [75.79805204646428]
  This paper presents a novel approach to Multi-Agent Reinforcement Learning (MARL) It combines cooperative task decomposition with the learning of reward machines (RMs) encoding the structure of the sub-tasks. The proposed method helps deal with the non-Markovian nature of the rewards in partially observable environments.
  arXiv  Detail & Related papers  (2023-03-24T15:12:28Z)
• Adaptive Value Decomposition with Greedy Marginal Contribution Computation for Cooperative Multi-Agent Reinforcement Learning [48.41925886860991]
  Real-world cooperation often requires intensive coordination among agents simultaneously. Traditional methods that learn the value function as a monotonic mixing of per-agent utilities cannot solve the tasks with non-monotonic returns. We propose a novel explicit credit assignment method to address the non-monotonic problem.
  arXiv  Detail & Related papers  (2023-02-14T07:23:59Z)
• Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning [2.9360071145551068]
  We propose a novel cooperative MARL algorithm named as interactive actor-critic(IAC) IAC models the interaction of agents from perspectives of policy and value function. We extend the value decomposition methods to continuous control tasks and evaluate IAC on benchmark tasks including classic control and multi-agent particle environments.
  arXiv  Detail & Related papers  (2021-02-10T01:58:28Z)
• UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learning [53.73686229912562]
  We propose a novel MARL approach called Universal Value Exploration (UneVEn) UneVEn learns a set of related tasks simultaneously with a linear decomposition of universal successor features. Empirical results on a set of exploration games, challenging cooperative predator-prey tasks requiring significant coordination among agents, and StarCraft II micromanagement benchmarks show that UneVEn can solve tasks where other state-of-the-art MARL methods fail.
  arXiv  Detail & Related papers  (2020-10-06T19:08:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.