What deep reinforcement learning tells us about human motor learning and
vice-versa
- URL: http://arxiv.org/abs/2208.10892v1
- Date: Tue, 23 Aug 2022 11:56:49 GMT
- Title: What deep reinforcement learning tells us about human motor learning and
vice-versa
- Authors: Michele Garibbo, Casimir Ludwig, Nathan Lepora and Laurence Aitchison
- Abstract summary: We show how recent deep RL methods correspond to the dominant motor learning framework in neuroscience, error-based learning.
We introduce a novel deep RL algorithm: model-based deterministic policy gradients (MB-DPG)
MB-DPG draws inspiration from error-based learning by explicitly relying on the observed outcome of actions.
- Score: 24.442174952832108
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Machine learning and specifically reinforcement learning (RL) has been
extremely successful in helping us to understand neural decision making
processes. However, RL's role in understanding other neural processes
especially motor learning is much less well explored. To explore this
connection, we investigated how recent deep RL methods correspond to the
dominant motor learning framework in neuroscience, error-based learning.
Error-based learning can be probed using a mirror reversal adaptation paradigm,
where it produces distinctive qualitative predictions that are observed in
humans. We therefore tested three major families of modern deep RL algorithm on
a mirror reversal perturbation. Surprisingly, all of the algorithms failed to
mimic human behaviour and indeed displayed qualitatively different behaviour
from that predicted by error-based learning. To fill this gap, we introduce a
novel deep RL algorithm: model-based deterministic policy gradients (MB-DPG).
MB-DPG draws inspiration from error-based learning by explicitly relying on the
observed outcome of actions. We show MB-DPG captures (human) error-based
learning under mirror-reversal and rotational perturbation. Next, we
demonstrate error-based learning in the form of MB-DPG learns faster than
canonical model-free algorithms on complex arm-based reaching tasks, while
being more robust to (forward) model misspecification than model-based RL.
These findings highlight the gap between current deep RL methods and human
motor adaptation and offer a route to closing this gap, facilitating future
beneficial interaction between between the two fields.
Related papers
- Implicit Offline Reinforcement Learning via Supervised Learning [83.8241505499762]
Offline Reinforcement Learning (RL) via Supervised Learning is a simple and effective way to learn robotic skills from a dataset collected by policies of different expertise levels.
We show how implicit models can leverage return information and match or outperform explicit algorithms to acquire robotic skills from fixed datasets.
arXiv Detail & Related papers (2022-10-21T21:59:42Z) - Entropy Regularized Reinforcement Learning with Cascading Networks [9.973226671536041]
Deep RL uses neural networks as function approximators.
One of the major difficulties of RL is the absence of i.i.d. data.
In this work, we challenge the common practices of the (un)supervised learning community of using a fixed neural architecture.
arXiv Detail & Related papers (2022-10-16T10:28:59Z) - Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning [92.18524491615548]
Contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL)
We study how RL can be empowered by contrastive learning in a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions.
Under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs.
arXiv Detail & Related papers (2022-07-29T17:29:08Z) - Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
There is an increasing need for active learning algorithms that are compatible with deep neural networks.
This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
arXiv Detail & Related papers (2021-06-17T17:26:31Z) - On the Reliability and Generalizability of Brain-inspired Reinforcement
Learning Algorithms [10.09712608508383]
We show that the computational model combining model-based and model-free control, which we term the prefrontal RL, reliably encodes the information of high-level policy that humans learned.
This is the first attempt to formally test the possibility that computational models mimicking the way the brain solves general problems can lead to practical solutions.
arXiv Detail & Related papers (2020-07-09T06:32:42Z) - Efficient Model-Based Reinforcement Learning through Optimistic Policy
Search and Planning [93.1435980666675]
We show how optimistic exploration can be easily combined with state-of-the-art reinforcement learning algorithms.
Our experiments demonstrate that optimistic exploration significantly speeds-up learning when there are penalties on actions.
arXiv Detail & Related papers (2020-06-15T18:37:38Z) - Stealing Deep Reinforcement Learning Models for Fun and Profit [33.64948529132546]
This paper presents the first model extraction attack against Deep Reinforcement Learning (DRL)
It enables an external adversary to precisely recover a black-box DRL model only from its interaction with the environment.
arXiv Detail & Related papers (2020-06-09T03:24:35Z) - AutoML-Zero: Evolving Machine Learning Algorithms From Scratch [76.83052807776276]
We show that it is possible to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks.
We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space.
We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction in the field.
arXiv Detail & Related papers (2020-03-06T19:00:04Z) - Information Theoretic Model Predictive Q-Learning [64.74041985237105]
We present a novel theoretical connection between information theoretic MPC and entropy regularized RL.
We develop a Q-learning algorithm that can leverage biased models.
arXiv Detail & Related papers (2019-12-31T00:29:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.