Concept Learning for Interpretable Multi-Agent Reinforcement Learning
- URL: http://arxiv.org/abs/2302.12232v1
- Date: Thu, 23 Feb 2023 18:53:09 GMT
- Title: Concept Learning for Interpretable Multi-Agent Reinforcement Learning
- Authors: Renos Zabounidis, Joseph Campbell, Simon Stepputtis, Dana Hughes,
Katia Sycara
- Abstract summary: We introduce a method for incorporating interpretable concepts from a domain expert into models trained through multi-agent reinforcement learning.
This allows an expert to both reason about the resulting concept policy models in terms of these high-level concepts at run-time, as well as intervene and correct mispredictions to improve performance.
We show that this yields improved interpretability and training stability, with benefits to policy performance and sample efficiency in a simulated and real-world cooperative-competitive multi-agent game.
- Score: 5.179808182296037
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multi-agent robotic systems are increasingly operating in real-world
environments in close proximity to humans, yet are largely controlled by policy
models with inscrutable deep neural network representations. We introduce a
method for incorporating interpretable concepts from a domain expert into
models trained through multi-agent reinforcement learning, by requiring the
model to first predict such concepts then utilize them for decision making.
This allows an expert to both reason about the resulting concept policy models
in terms of these high-level concepts at run-time, as well as intervene and
correct mispredictions to improve performance. We show that this yields
improved interpretability and training stability, with benefits to policy
performance and sample efficiency in a simulated and real-world
cooperative-competitive multi-agent game.
Related papers
- Multi-agent Off-policy Actor-Critic Reinforcement Learning for Partially Observable Environments [30.280532078714455]
This study proposes the use of a social learning method to estimate a global state within a multi-agent off-policy actor-critic algorithm for reinforcement learning.
We show that the difference between final outcomes, obtained when the global state is fully observed versus estimated through the social learning method, is $varepsilon$-bounded when an appropriate number of iterations of social learning updates are implemented.
arXiv Detail & Related papers (2024-07-06T06:51:14Z) - Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment [69.33930972652594]
We propose a novel structural pruning approach to jointly learn the weights and structurally prune architectures of CNN models.
The core element of our method is a Reinforcement Learning (RL) agent whose actions determine the pruning ratios of the CNN model's layers.
We conduct the joint training and pruning by iteratively training the model's weights and the agent's policy.
arXiv Detail & Related papers (2024-03-28T15:22:29Z) - Contrastive learning-based agent modeling for deep reinforcement
learning [31.293496061727932]
Agent modeling is essential when designing adaptive policies for intelligent machine agents in multiagent systems.
We devised a Contrastive Learning-based Agent Modeling (CLAM) method that relies only on the local observations from the ego agent during training and execution.
CLAM is capable of generating consistent high-quality policy representations in real-time right from the beginning of each episode.
arXiv Detail & Related papers (2023-12-30T03:44:12Z) - Discovering How Agents Learn Using Few Data [32.38609641970052]
We propose a theoretical and algorithmic framework for real-time identification of agent behavior using a short burst of a single system trajectory.
Our approach accurately recovers the true dynamics across various benchmarks, including equilibrium selection and prediction of chaotic systems up to 10 Lynov times.
These findings suggest that our approach has significant potential to support effective policy and decision-making in strategic multi-agent systems.
arXiv Detail & Related papers (2023-07-13T09:14:48Z) - Predictive Experience Replay for Continual Visual Control and
Forecasting [62.06183102362871]
We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting.
We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting.
Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
arXiv Detail & Related papers (2023-03-12T05:08:03Z) - Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent
Reinforcement Learning [15.12491397254381]
We propose an implicit model-based multi-agent reinforcement learning method based on value decomposition methods.
Under this method, agents can interact with the learned virtual environment and evaluate the current state value according to imagined future states.
arXiv Detail & Related papers (2022-04-20T12:16:27Z) - Multi-Agent Imitation Learning with Copulas [102.27052968901894]
Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions.
In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems.
Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
arXiv Detail & Related papers (2021-07-10T03:49:41Z) - Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning [89.31889875864599]
We propose an efficient model-based reinforcement learning algorithm for learning in multi-agent systems.
Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC.
We provide a practical parametrization of the core optimization problem.
arXiv Detail & Related papers (2021-07-08T18:01:02Z) - Deep Interactive Bayesian Reinforcement Learning via Meta-Learning [63.96201773395921]
The optimal adaptive behaviour under uncertainty over the other agents' strategies can be computed using the Interactive Bayesian Reinforcement Learning framework.
We propose to meta-learn approximate belief inference and Bayes-optimal behaviour for a given prior.
We show empirically that our approach outperforms existing methods that use a model-free approach, sample from the approximate posterior, maintain memory-free models of others, or do not fully utilise the known structure of the environment.
arXiv Detail & Related papers (2021-01-11T13:25:13Z) - Learning Diverse Representations for Fast Adaptation to Distribution
Shift [78.83747601814669]
We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task.
We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
arXiv Detail & Related papers (2020-06-12T12:23:50Z) - Improving Robot Dual-System Motor Learning with Intrinsically Motivated
Meta-Control and Latent-Space Experience Imagination [17.356402088852423]
We present a novel dual-system motor learning approach where a meta-controller arbitrates online between model-based and model-free decisions.
We evaluate our approach against baseline and state-of-the-art methods on learning vision-based robotic grasping in simulation and real world.
arXiv Detail & Related papers (2020-04-19T12:14:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.