Recursive Least Squares Policy Control with Echo State Network

• URL: http://arxiv.org/abs/2201.04781v1
• Date: Thu, 13 Jan 2022 03:41:07 GMT
• Title: Recursive Least Squares Policy Control with Echo State Network
• Authors: Chunyuan Zhang, Chao Liu, Qi Song and Jie Zhao
• Abstract summary: We propose two novel policy control algorithms, ESNRLS-Q and ESNRLS-Sarsa. To reduce the correlation of training samples, we use the leaky integrator ESN and the mini-batch learning mode. To make RLS suitable for training ESN in mini-batch mode, we present a new mean-approximation method for updating the RLS correlation matrix.
• Score: 17.555929738017344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The echo state network (ESN) is a special type of recurrent neural networks for processing the time-series dataset. However, limited by the strong correlation among sequential samples of the agent, ESN-based policy control algorithms are difficult to use the recursive least squares (RLS) algorithm to update the ESN's parameters. To solve this problem, we propose two novel policy control algorithms, ESNRLS-Q and ESNRLS-Sarsa. Firstly, to reduce the correlation of training samples, we use the leaky integrator ESN and the mini-batch learning mode. Secondly, to make RLS suitable for training ESN in mini-batch mode, we present a new mean-approximation method for updating the RLS correlation matrix. Thirdly, to prevent ESN from over-fitting, we use the L1 regularization technique. Lastly, to prevent the target state-action value from overestimation, we employ the Mellowmax method. Simulation results show that our algorithms have good convergence performance.

Related papers

• Recurrent Stochastic Configuration Networks for Temporal Data Analytics [3.8719670789415925]
  This paper develops a recurrent version of configuration networks (RSCNs) for problem solving. We build an initial RSCN model in the light of a supervisory mechanism, followed by an online update of the output weights. Numerical results clearly indicate that the proposed RSCN performs favourably over all of the datasets.
  arXiv  Detail & Related papers  (2024-06-21T03:21:22Z)
• Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding [57.71603937699949]
  We study optimization guarantees, i.e., achieving near-zero training loss with the increase in the number of learning epochs. We show that the threshold on the number of training samples increases with the increase in the network width.
  arXiv  Detail & Related papers  (2023-09-12T13:03:47Z)
• A Comparative Study of Deep Learning and Iterative Algorithms for Joint Channel Estimation and Signal Detection in OFDM Systems [11.190815358585137]
  Joint channel estimation and signal detection is crucial in frequency division multiplexing systems. Traditional algorithms perform poorly in low signal-to-noise ratio (SNR) scenarios. Deep learning (DL) methods have been investigated, but concerns regarding computational expense and lack of validation in low-SNR settings remain.
  arXiv  Detail & Related papers  (2023-03-07T06:34:04Z)
• Recursive Least Squares Advantage Actor-Critic Algorithms [20.792917267835247]
  We propose two novel RLS-based advantage actor critic (A2C) algorithms. RLSSA2C and RLSNA2C, use the RLS method to train the critic network and the hidden layers of the actor network. From the experimental results, it is shown that our both algorithms have better sample efficiency than the vanilla A2C on most games or tasks.
  arXiv  Detail & Related papers  (2022-01-15T20:00:26Z)
• Verification of Neural-Network Control Systems by Integrating Taylor Models and Zonotopes [0.0]
  We study the verification problem for closed-loop dynamical systems with neural-network controllers (NNCS) We present an algorithm to chain approaches based on Taylor models and zonotopes, yielding a precise reachability algorithm for NNCS.
  arXiv  Detail & Related papers  (2021-12-16T20:46:39Z)
• Robust lEarned Shrinkage-Thresholding (REST): Robust unrolling for sparse recover [87.28082715343896]
  We consider deep neural networks for solving inverse problems that are robust to forward model mis-specifications. We design a new robust deep neural network architecture by applying algorithm unfolding techniques to a robust version of the underlying recovery problem. The proposed REST network is shown to outperform state-of-the-art model-based and data-driven algorithms in both compressive sensing and radar imaging problems.
  arXiv  Detail & Related papers  (2021-10-20T06:15:45Z)
• Revisiting Recursive Least Squares for Training Deep Neural Networks [10.44340837533087]
  Recursive least squares (RLS) algorithms were once widely used for training small-scale neural networks, due to their fast convergence. Previous RLS algorithms are unsuitable for training deep neural networks (DNNs), since they have high computational complexity and too many preconditions. We propose three novel RLS optimization algorithms for training feedforward neural networks, convolutional neural networks and recurrent neural networks.
  arXiv  Detail & Related papers  (2021-09-07T17:43:51Z)
• LocalDrop: A Hybrid Regularization for Deep Neural Networks [98.30782118441158]
  We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs) has been developed based on the proposed upper bound of the local Rademacher complexity.
  arXiv  Detail & Related papers  (2021-03-01T03:10:11Z)
• Deep Networks for Direction-of-Arrival Estimation in Low SNR [89.45026632977456]
  We introduce a Convolutional Neural Network (CNN) that is trained from mutli-channel data of the true array manifold matrix. We train a CNN in the low-SNR regime to predict DoAs across all SNRs. Our robust solution can be applied in several fields, ranging from wireless array sensors to acoustic microphones or sonars.
  arXiv  Detail & Related papers  (2020-11-17T12:52:18Z)
• Continual Learning in Recurrent Neural Networks [67.05499844830231]
  We evaluate the effectiveness of continual learning methods for processing sequential data with recurrent neural networks (RNNs) We shed light on the particularities that arise when applying weight-importance methods, such as elastic weight consolidation, to RNNs. We show that the performance of weight-importance methods is not directly affected by the length of the processed sequences, but rather by high working memory requirements.
  arXiv  Detail & Related papers  (2020-06-22T10:05:12Z)
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks [50.42141893913188]
  We study a distributed variable for large-scale AUC for a neural network as with a deep neural network. Our model requires a much less number of communication rounds and still a number of communication rounds in theory. Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
  arXiv  Detail & Related papers  (2020-05-05T18:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.