Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks

• URL: http://arxiv.org/abs/2003.07417v1
• Date: Mon, 16 Mar 2020 19:21:08 GMT
• Title: Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks
• Authors: Sina Ghiassian, Banafsheh Rafiee, Yat Long Lo, Adam White
• Abstract summary: We show how online NN training and interference interact in reinforcement learning. We find that simply re-mapping the input observations to a high-dimensional space improves learning speed and parameter sensitivity. We provide a simple approach to NN training that is easy to implement, and requires little additional computation.
• Score: 5.273501657421096
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning systems require good representations to work well. For decades practical success in reinforcement learning was limited to small domains. Deep reinforcement learning systems, on the other hand, are scalable, not dependent on domain specific prior knowledge and have been successfully used to play Atari, in 3D navigation from pixels, and to control high degree of freedom robots. Unfortunately, the performance of deep reinforcement learning systems is sensitive to hyper-parameter settings and architecture choices. Even well tuned systems exhibit significant instability both within a trial and across experiment replications. In practice, significant expertise and trial and error are usually required to achieve good performance. One potential source of the problem is known as catastrophic interference: when later training decreases performance by overriding previous learning. Interestingly, the powerful generalization that makes Neural Networks (NN) so effective in batch supervised learning might explain the challenges when applying them in reinforcement learning tasks. In this paper, we explore how online NN training and interference interact in reinforcement learning. We find that simply re-mapping the input observations to a high-dimensional space improves learning speed and parameter sensitivity. We also show this preprocessing reduces interference in prediction tasks. More practically, we provide a simple approach to NN training that is easy to implement, and requires little additional computation. We demonstrate that our approach improves performance in both prediction and control with an extensive batch of experiments in classic control domains.

Related papers

• Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification [0.0]
  This paper presents a dynamic learning rate scheduler (DLRS) algorithm that adapts the learning rate based on the loss values calculated during the training process.<n> Experiments are conducted on problems related to physics-informed neural networks (PINNs) and image classification using multilayer perceptrons and convolutional neural networks, respectively.
  arXiv  Detail & Related papers  (2025-07-29T12:31:21Z)
• What Can Grokking Teach Us About Learning Under Nonstationarity? [21.031486400628854]
  In continual learning problems, it is necessary to overwrite components of a neural network's learned representation in response to changes in the data stream.<n> neural networks often exhibit primacy bias, whereby early training data hinders the network's ability to generalize on later tasks.<n>We show that the emergence of feature-learning dynamics is known to drive the phenomenon of grokking.
  arXiv  Detail & Related papers  (2025-07-26T20:51:24Z)
• Normalization and effective learning rates in reinforcement learning [52.59508428613934]
  Normalization layers have recently experienced a renaissance in the deep reinforcement learning and continual learning literature. We show that normalization brings with it a subtle but important side effect: an equivalence between growth in the norm of the network parameters and decay in the effective learning rate. We propose to make the learning rate schedule explicit with a simple re- parameterization which we call Normalize-and-Project.
  arXiv  Detail & Related papers  (2024-07-01T20:58:01Z)
• REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation [61.7171775202833]
  We introduce an efficient system for learning dexterous manipulation skills withReinforcement learning. The main idea of our approach is the integration of recent advances in sample-efficient RL and replay buffer bootstrapping. Our system completes the real-world training cycle by incorporating learned resets via an imitation-based pickup policy.
  arXiv  Detail & Related papers  (2023-09-06T19:05:31Z)
• Adversarial Training Using Feedback Loops [1.6114012813668932]
  Deep neural networks (DNNs) are highly susceptible to adversarial attacks due to limited generalizability. This paper proposes a new robustification approach based on control theory. The novel adversarial training approach based on the feedback control architecture is called Feedback Looped Adversarial Training (FLAT)
  arXiv  Detail & Related papers  (2023-08-23T02:58:02Z)
• Solving Large-scale Spatial Problems with Convolutional Neural Networks [88.31876586547848]
  We employ transfer learning to improve training efficiency for large-scale spatial problems. We propose that a convolutional neural network (CNN) can be trained on small windows of signals, but evaluated on arbitrarily large signals with little to no performance degradation.
  arXiv  Detail & Related papers  (2023-06-14T01:24:42Z)
• The least-control principle for learning at equilibrium [65.2998274413952]
  We present a new principle for learning equilibrium recurrent neural networks, deep equilibrium models, or meta-learning. Our results shed light on how the brain might learn and offer new ways of approaching a broad class of machine learning problems.
  arXiv  Detail & Related papers  (2022-07-04T11:27:08Z)
• Hebbian Continual Representation Learning [9.54473759331265]
  Continual Learning aims to bring machine learning into a more realistic scenario. We investigate whether biologically inspired Hebbian learning is useful for tackling continual challenges.
  arXiv  Detail & Related papers  (2022-06-28T09:21:03Z)
• Improving the sample-efficiency of neural architecture search with reinforcement learning [0.0]
  In this work, we would like to contribute to the area of Automated Machine Learning (AutoML) Our focus is on one of the most promising research directions, reinforcement learning. The validation accuracies of the child networks serve as a reward signal for training the controller. We propose to modify this to a more modern and complex algorithm, PPO, which has demonstrated to be faster and more stable in other environments.
  arXiv  Detail & Related papers  (2021-10-13T14:30:09Z)
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings [89.63764845984076]
  We present Stored Embeddings for Efficient Reinforcement Learning (SEER) SEER is a simple modification of existing off-policy deep reinforcement learning methods. We show that SEER does not degrade the performance of RLizable agents while significantly saving computation and memory.
  arXiv  Detail & Related papers  (2021-03-04T08:14:10Z)
• Improving Learning Efficiency for Wireless Resource Allocation with Symmetric Prior [28.275250620630466]
  In this article, we first briefly summarize two classes of approaches to using domain knowledge: introducing mathematical models or prior knowledge to deep learning. To explain how such a generic prior is harnessed to improve learning efficiency, we resort to ranking. We find that the required training samples to achieve given system performance decreases with the number of subcarriers or contents.
  arXiv  Detail & Related papers  (2020-05-18T07:57:34Z)
• The large learning rate phase of deep learning: the catapult mechanism [50.23041928811575]
  We present a class of neural networks with solvable training dynamics. We find good agreement between our model's predictions and training dynamics in realistic deep learning settings. We believe our results shed light on characteristics of models trained at different learning rates.
  arXiv  Detail & Related papers  (2020-03-04T17:52:48Z)
• Memristor Hardware-Friendly Reinforcement Learning [14.853739554366351]
  We propose a memristive neuromorphic hardware implementation for the actor-critic algorithm in reinforcement learning. We consider the task of balancing an inverted pendulum, a classical problem in both RL and control theory. We believe that this study shows the promise of using memristor-based hardware neural networks for handling complex tasks through in-situ reinforcement learning.
  arXiv  Detail & Related papers  (2020-01-20T01:08:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.