Privacy-Engineered Value Decomposition Networks for Cooperative
Multi-Agent Reinforcement Learning
- URL: http://arxiv.org/abs/2311.06255v1
- Date: Wed, 13 Sep 2023 02:50:57 GMT
- Title: Privacy-Engineered Value Decomposition Networks for Cooperative
Multi-Agent Reinforcement Learning
- Authors: Parham Gohari, Matthew Hale, and Ufuk Topcu
- Abstract summary: In cooperative multi-agent reinforcement learning, a team of agents must jointly optimize the team's long-term rewards to learn a designated task.
Privacy-Engineered Value Decomposition Networks (PE-VDN) models multi-agent coordination while safeguarding the confidentiality of the agents' environment interaction data.
We implement PE-VDN in StarCraft Multi-Agent Competition (SMAC) and show that it achieves 80% of Vanilla VDN's win rate.
- Score: 19.504842607744457
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In cooperative multi-agent reinforcement learning (Co-MARL), a team of agents
must jointly optimize the team's long-term rewards to learn a designated task.
Optimizing rewards as a team often requires inter-agent communication and data
sharing, leading to potential privacy implications. We assume privacy
considerations prohibit the agents from sharing their environment interaction
data. Accordingly, we propose Privacy-Engineered Value Decomposition Networks
(PE-VDN), a Co-MARL algorithm that models multi-agent coordination while
provably safeguarding the confidentiality of the agents' environment
interaction data. We integrate three privacy-engineering techniques to redesign
the data flows of the VDN algorithm, an existing Co-MARL algorithm that
consolidates the agents' environment interaction data to train a central
controller that models multi-agent coordination, and develop PE-VDN. In the
first technique, we design a distributed computation scheme that eliminates
Vanilla VDN's dependency on sharing environment interaction data. Then, we
utilize a privacy-preserving multi-party computation protocol to guarantee that
the data flows of the distributed computation scheme do not pose new privacy
risks. Finally, we enforce differential privacy to preempt inference threats
against the agents' training data, past environment interactions, when they
take actions based on their neural network predictions. We implement PE-VDN in
StarCraft Multi-Agent Competition (SMAC) and show that it achieves 80% of
Vanilla VDN's win rate while maintaining differential privacy levels that
provide meaningful privacy guarantees. The results demonstrate that PE-VDN can
safeguard the confidentiality of agents' environment interaction data without
sacrificing multi-agent coordination.
Related papers
- Privacy Preserving Semi-Decentralized Mean Estimation over Intermittently-Connected Networks [59.43433767253956]
We consider the problem of privately estimating the mean of vectors distributed across different nodes of an unreliable wireless network.
In a semi-decentralized setup, nodes can collaborate with their neighbors to compute a local consensus, which they relay to a central server.
We study the tradeoff between collaborative relaying and privacy leakage due to the data sharing among nodes.
arXiv Detail & Related papers (2024-06-06T06:12:15Z) - Binary Federated Learning with Client-Level Differential Privacy [7.854806519515342]
Federated learning (FL) is a privacy-preserving collaborative learning framework.
Existing FL systems typically adopt Federated Average (FedAvg) as the training algorithm.
We propose a communication-efficient FL training algorithm with differential privacy guarantee.
arXiv Detail & Related papers (2023-08-07T06:07:04Z) - Differentially Private Decentralized Deep Learning with Consensus
Algorithms [10.208363125551555]
Cooperative decentralized deep learning relies on direct information exchange between communicating agents.
Sharing parameters with untrustworthy neighboring agents could leak exploitable information about local datasets.
We introduce differentially private decentralized learning that secures each agent's local dataset during and after cooperative training.
arXiv Detail & Related papers (2023-06-24T07:46:00Z) - PS-FedGAN: An Efficient Federated Learning Framework Based on Partially
Shared Generative Adversarial Networks For Data Privacy [56.347786940414935]
Federated Learning (FL) has emerged as an effective learning paradigm for distributed computation.
This work proposes a novel FL framework that requires only partial GAN model sharing.
Named as PS-FedGAN, this new framework enhances the GAN releasing and training mechanism to address heterogeneous data distributions.
arXiv Detail & Related papers (2023-05-19T05:39:40Z) - Collaborative Mean Estimation over Intermittently Connected Networks
with Peer-To-Peer Privacy [86.61829236732744]
This work considers the problem of Distributed Mean Estimation (DME) over networks with intermittent connectivity.
The goal is to learn a global statistic over the data samples localized across distributed nodes with the help of a central server.
We study the tradeoff between collaborative relaying and privacy leakage due to the additional data sharing among nodes.
arXiv Detail & Related papers (2023-02-28T19:17:03Z) - Privacy-Preserving Joint Edge Association and Power Optimization for the
Internet of Vehicles via Federated Multi-Agent Reinforcement Learning [74.53077322713548]
We investigate the privacy-preserving joint edge association and power allocation problem.
The proposed solution strikes a compelling trade-off, while preserving a higher privacy level than the state-of-the-art solutions.
arXiv Detail & Related papers (2023-01-26T10:09:23Z) - Secure Distributed/Federated Learning: Prediction-Privacy Trade-Off for
Multi-Agent System [4.190359509901197]
In the big data era, performing inference within the distributed and federated learning (DL and FL) frameworks, the central server needs to process a large amount of data.
Considering the decentralized computing topology, privacy has become a first-class concern.
We study the textitprivacy-aware server to multi-agent assignment problem subject to information processing constraints associated with each agent.
arXiv Detail & Related papers (2022-04-24T19:19:20Z) - Mitigating Leakage from Data Dependent Communications in Decentralized
Computing using Differential Privacy [1.911678487931003]
We propose a general execution model to control the data-dependence of communications in user-side decentralized computations.
Our formal privacy guarantees leverage and extend recent results on privacy amplification by shuffling.
arXiv Detail & Related papers (2021-12-23T08:30:17Z) - Cooperative Multi-Agent Actor-Critic for Privacy-Preserving Load
Scheduling in a Residential Microgrid [71.17179010567123]
We propose a privacy-preserving multi-agent actor-critic framework where the decentralized actors are trained with distributed critics.
The proposed framework can preserve the privacy of the households while simultaneously learn the multi-agent credit assignment mechanism implicitly.
arXiv Detail & Related papers (2021-10-06T14:05:26Z) - Learning Connectivity for Data Distribution in Robot Teams [96.39864514115136]
We propose a task-agnostic, decentralized, low-latency method for data distribution in ad-hoc networks using Graph Neural Networks (GNN)
Our approach enables multi-agent algorithms based on global state information to function by ensuring it is available at each robot.
We train the distributed GNN communication policies via reinforcement learning using the average Age of Information as the reward function and show that it improves training stability compared to task-specific reward functions.
arXiv Detail & Related papers (2021-03-08T21:48:55Z) - PRICURE: Privacy-Preserving Collaborative Inference in a Multi-Party
Setting [3.822543555265593]
This paper presents PRICURE, a system that combines complementary strengths of secure multi-party computation and differential privacy.
PRICURE enables privacy-preserving collaborative prediction among multiple model owners.
We evaluate PRICURE on neural networks across four datasets including benchmark medical image classification datasets.
arXiv Detail & Related papers (2021-02-19T05:55:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.