Brain-Like Replay Naturally Emerges in Reinforcement Learning Agents

• URL: http://arxiv.org/abs/2402.01467v2
• Date: Sun, 06 Oct 2024 21:37:54 GMT
• Title: Brain-Like Replay Naturally Emerges in Reinforcement Learning Agents
• Authors: Jiyi Wang, Likai Tang, Huimiao Chen, Marcelo G Mattar, Sen Song,
• Abstract summary: We develop a modular reinforcement learning model that could generate replay. We prove that replay generated in this way helps complete the task. Our design avoids complex assumptions and enables replay to emerge naturally within a task-optimized paradigm.
• Score: 3.9276584971242303
• License:
• Abstract: Replay is a powerful strategy to promote learning in artificial intelligence and the brain. However, the conditions to generate it and its functional advantages have not been fully recognized. In this study, we develop a modular reinforcement learning model that could generate replay. We prove that replay generated in this way helps complete the task. We also analyze the information contained in the representation and provide a mechanism for how replay makes a difference. Our design avoids complex assumptions and enables replay to emerge naturally within a task-optimized paradigm. Our model also reproduces key phenomena observed in biological agents. This research explores the structural biases in modular ANN to generate replay and its potential utility in developing efficient RL.

Related papers

• Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning [69.19840497497503]
  It is argued that the commonly used action matching principle is more like an explanation of deep neural networks (DNNs) than the interpretation of RL agents. We propose to consider rewards, the essential objective of RL agents, as the essential objective of interpreting RL agents. We verify and evaluate our method on the Atari 2600 games as well as Duckietown, a challenging self-driving car simulator environment.
  arXiv  Detail & Related papers  (2023-09-04T09:09:54Z)
• Adiabatic replay for continual learning [138.7878582237908]
  generative replay spends an increasing amount of time just re-learning what is already known. We propose a replay-based CL strategy that we term adiabatic replay (AR) We verify experimentally that AR is superior to state-of-the-art deep generative replay using VAEs.
  arXiv  Detail & Related papers  (2023-03-23T10:18:06Z)
• Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 [5.239932780277599]
  Generative replay (GR) is a biologically-inspired replay mechanism that augments learning experiences with self-labelled examples. We present a version of GR for LRL that satisfies two desideratas: (a) Introspective density modelling of the latent representations of policies learned using deep RL, and (b) Model-free end-to-end learning.
  arXiv  Detail & Related papers  (2022-08-09T22:00:28Z)
• Architecting and Visualizing Deep Reinforcement Learning Models [77.34726150561087]
  Deep Reinforcement Learning (DRL) is a theory that aims to teach computers how to communicate with each other. In this paper, we present a new Atari Pong game environment, a policy gradient based DRL model, a real-time network visualization, and an interactive display to help build intuition and awareness of the mechanics of DRL inference.
  arXiv  Detail & Related papers  (2021-12-02T17:48:26Z)
• Learning offline: memory replay in biological and artificial reinforcement learning [1.0136215038345011]
  We review the functional roles of replay in the fields of neuroscience and AI. Replay is important for memory consolidation in biological neural networks. It is also key to stabilising learning in deep neural networks.
  arXiv  Detail & Related papers  (2021-09-21T08:57:19Z)
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning [27.593497502386143]
  Theory-Based Reinforcement Learning uses human-like intuitive theories to explore and model an environment. We instantiate the approach in a video game playing agent called EMPA. EMPA matches human learning efficiency on a suite of 90 Atari-style video games.
  arXiv  Detail & Related papers  (2021-07-27T01:38:13Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)
• Replay in Deep Learning: Current Approaches and Missing Biological Elements [33.20770284464084]
  Replay is the reactivation of one or more neural patterns. It is thought to play a critical role in memory formation, retrieval, and consolidation. We provide the first comprehensive comparison between replay in the mammalian brain and replay in artificial neural networks.
  arXiv  Detail & Related papers  (2021-04-01T15:19:08Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations [55.41644538483948]
  In this paper, we propose a combination of approaches that allow the agent to use low-quality demonstrations in complex vision-based environments. Our proposed goal-oriented structuring of replay buffer allows the agent to automatically highlight sub-goals for solving complex hierarchical tasks in demonstrations. The solution based on our algorithm beats all the solutions for the famous MineRL competition and allows the agent to mine a diamond in the Minecraft environment.
  arXiv  Detail & Related papers  (2020-06-17T15:38:40Z)
• Dynamic Experience Replay [6.062589413216726]
  We build upon Ape-X DDPG and demonstrate our approach on robotic tight-fitting joint assembly tasks. In particular, we run experiments on two different tasks: peg-in-hole and lap-joint. Our ablation studies show that Dynamic Experience Replay is a crucial ingredient that either largely shortens the training time in these challenging environments.
  arXiv  Detail & Related papers  (2020-03-04T23:46:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.