Real-Time Recurrent Reinforcement Learning

• URL: http://arxiv.org/abs/2311.04830v2
• Date: Thu, 28 Mar 2024 10:30:57 GMT
• Title: Real-Time Recurrent Reinforcement Learning
• Authors: Julian Lemmel, Radu Grosu,
• Abstract summary: RTRRL consists of three parts: (1) a Meta-RL RNN architecture, implementing on its own an actor-critic algorithm; (2) an outer reinforcement learning algorithm, exploiting temporal difference learning and dutch eligibility traces to train the Meta-RL network; and (3) random-feedback local-online (RFLO) learning, an online automatic differentiation algorithm for computing the gradients with respect to parameters of the network.
• Score: 7.737685867200335
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper we propose real-time recurrent reinforcement learning (RTRRL), a biologically plausible approach to solving discrete and continuous control tasks in partially-observable markov decision processes (POMDPs). RTRRL consists of three parts: (1) a Meta-RL RNN architecture, implementing on its own an actor-critic algorithm; (2) an outer reinforcement learning algorithm, exploiting temporal difference learning and dutch eligibility traces to train the Meta-RL network; and (3) random-feedback local-online (RFLO) learning, an online automatic differentiation algorithm for computing the gradients with respect to parameters of the network.Our experimental results show that by replacing the optimization algorithm in RTRRL with the biologically implausible back propagation through time (BPTT), or real-time recurrent learning (RTRL), one does not improve returns, while matching the computational complexity for BPTT, and even increasing complexity for RTRL. RTRRL thus serves as a model of learning in biological neural networks, mimicking reward pathways in the basal ganglia.

Related papers

• Real-Time Recurrent Learning using Trace Units in Reinforcement Learning [27.250024431890477]
  Recurrent Neural Networks (RNNs) are used to learn representations in partially observable environments. For agents that learn online and continually interact with the environment, it is desirable to train RNNs with real-time recurrent learning (RTRL) We build on these insights to provide a lightweight but effective approach for training RNNs in online RL.
  arXiv  Detail & Related papers  (2024-09-02T20:08:23Z)
• Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate [4.6659670917171825]
  Recurrent reinforcement learning (RL) consists of a context encoder based on recurrent neural networks (RNNs) for unobservable state prediction. Previous RL methods face training stability issues due to the gradient instability of RNNs. We propose Recurrent Off-policy RL with Context-Encoder-Specific Learning Rate (RESeL) to tackle this issue.
  arXiv  Detail & Related papers  (2024-05-24T09:33:47Z)
• Intelligent Hybrid Resource Allocation in MEC-assisted RAN Slicing Network [72.2456220035229]
  We aim to maximize the SSR for heterogeneous service demands in the cooperative MEC-assisted RAN slicing system. We propose a recurrent graph reinforcement learning (RGRL) algorithm to intelligently learn the optimal hybrid RA policy.
  arXiv  Detail & Related papers  (2024-05-02T01:36:13Z)
• Real-Time Progressive Learning: Accumulate Knowledge from Control with Neural-Network-Based Selective Memory [2.8638167607890836]
  A radial basis function neural network based learning control scheme named real-time progressive learning (RTPL) is proposed. RTPL learns unknown dynamics of the system with guaranteed stability and closed-loop performance.
  arXiv  Detail & Related papers  (2023-08-08T12:39:57Z)
• Properties and Potential Applications of Random Functional-Linked Types of Neural Networks [81.56822938033119]
  Random functional-linked neural networks (RFLNNs) offer an alternative way of learning in deep structure. This paper gives some insights into the properties of RFLNNs from the viewpoints of frequency domain. We propose a method to generate a BLS network with better performance, and design an efficient algorithm for solving Poison's equation.
  arXiv  Detail & Related papers  (2023-04-03T13:25:22Z)
• Efficient Real Time Recurrent Learning through combined activity and parameter sparsity [0.5076419064097732]
  Backpropagation through time (BPTT) is the standard algorithm for training recurrent neural networks (RNNs) BPTT is unsuited for online learning and presents a challenge for implementation on low-resource real-time systems. We show that recurrent networks exhibiting high activity sparsity can reduce the computational cost of Real-Time Recurrent Learning (RTRL)
  arXiv  Detail & Related papers  (2023-03-10T01:09:04Z)
• MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks [63.24965775030673]
  We propose a novel Reinforcement Learning (RL) approach to design generic Congestion Control (CC) algorithms. Our solution, MARLIN, uses the Soft Actor-Critic algorithm to maximize both entropy and return. We trained MARLIN on a real network with varying background traffic patterns to overcome the sim-to-real mismatch.
  arXiv  Detail & Related papers  (2023-02-02T18:27:20Z)
• ETLP: Event-based Three-factor Local Plasticity for online learning with neuromorphic hardware [105.54048699217668]
  We show a competitive performance in accuracy with a clear advantage in the computational complexity for Event-Based Three-factor Local Plasticity (ETLP) We also show that when using local plasticity, threshold adaptation in spiking neurons and a recurrent topology are necessary to learntemporal patterns with a rich temporal structure.
  arXiv  Detail & Related papers  (2023-01-19T19:45:42Z)
• Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning [92.18524491615548]
  Contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL) We study how RL can be empowered by contrastive learning in a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. Under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs.
  arXiv  Detail & Related papers  (2022-07-29T17:29:08Z)
• A Heuristically Assisted Deep Reinforcement Learning Approach for Network Slice Placement [0.7885276250519428]
  We introduce a hybrid placement solution based on Deep Reinforcement Learning (DRL) and a dedicated optimization based on the Power of Two Choices principle. The proposed Heuristically-Assisted DRL (HA-DRL) allows to accelerate the learning process and gain in resource usage when compared against other state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-05-14T10:04:17Z)
• Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning [56.17667147101263]
  In real-world tasks, reinforcement learning agents encounter situations that are not present during training time. To ensure reliable performance, the RL agents need to exhibit robustness against worst-case situations. We propose the Robust Hallucinated Upper-Confidence RL (RH-UCRL) algorithm to provably solve this problem.
  arXiv  Detail & Related papers  (2021-03-18T16:50:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.