Continuous Homeostatic Reinforcement Learning for Self-Regulated Autonomous Agents

• URL: http://arxiv.org/abs/2109.06580v1
• Date: Tue, 14 Sep 2021 11:03:58 GMT
• Title: Continuous Homeostatic Reinforcement Learning for Self-Regulated Autonomous Agents
• Authors: Hugo Lauren\c{c}on, Charbel-Rapha\"el S\'egerie, Johann Lussange, Boris S. Gutkin
• Abstract summary: We propose an extension of the homeostatic reinforcement learning theory to a continuous environment in space and time. Inspired by the self-regulating mechanisms abundantly present in biology, we also introduce a model for the dynamics of the agent internal state.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Homeostasis is a prevalent process by which living beings maintain their internal milieu around optimal levels. Multiple lines of evidence suggest that living beings learn to act to predicatively ensure homeostasis (allostasis). A classical theory for such regulation is drive reduction, where a function of the difference between the current and the optimal internal state. The recently introduced homeostatic regulated reinforcement learning theory (HRRL), by defining within the framework of reinforcement learning a reward function based on the internal state of the agent, makes the link between the theories of drive reduction and reinforcement learning. The HRRL makes it possible to explain multiple eating disorders. However, the lack of continuous change in the internal state of the agent with the discrete-time modeling has been so far a key shortcoming of the HRRL theory. Here, we propose an extension of the homeostatic reinforcement learning theory to a continuous environment in space and time, while maintaining the validity of the theoretical results and the behaviors explained by the model in discrete time. Inspired by the self-regulating mechanisms abundantly present in biology, we also introduce a model for the dynamics of the agent internal state, requiring the agent to continuously take actions to maintain homeostasis. Based on the Hamilton-Jacobi-Bellman equation and function approximation with neural networks, we derive a numerical scheme allowing the agent to learn directly how its internal mechanism works, and to choose appropriate action policies via reinforcement learning and an appropriate exploration of the environment. Our numerical experiments show that the agent does indeed learn to behave in a way that is beneficial to its survival in the environment, making our framework promising for modeling animal dynamics and decision-making.

Related papers

• Surprise! Using Physiological Stress for Allostatic Regulation Under the Active Inference Framework [Pre-Print] [0.5586191108738563]
  We develop a model that grounds prediction errors into the secretion of a physiological stress hormone (cortisol) acting as an adaptive mediator on a homeostatically-controlled physiology. Our results find that allostatic functions of cortisol (stress) provide adaptive advantages to the agent's long-term physiological regulation.
  arXiv  Detail & Related papers  (2024-06-12T17:56:15Z)
• Continuous Time Continuous Space Homeostatic Reinforcement Learning (CTCS-HRRL) : Towards Biological Self-Autonomous Agent [0.12068041242343093]
  Homeostasis is a process by which living beings maintain their internal balance. Homeostatic Regulated Reinforcement Learning (HRRL) framework attempts to explain this learned homeostatic behaviour. In this work, we advance the HRRL framework to a continuous time-space environment and validate the CTCS-HRRL framework.
  arXiv  Detail & Related papers  (2024-01-17T06:29:34Z)
• Emergence of Adaptive Circadian Rhythms in Deep Reinforcement Learning [0.0]
  Adapting to regularities of the environment is critical for biological organisms to anticipate events and plan. We study the emergence of circadian-like rhythms in deep reinforcement learning agents.
  arXiv  Detail & Related papers  (2023-07-22T18:47:18Z)
• Learning Goal-based Movement via Motivational-based Models in Cognitive Mobile Robots [58.720142291102135]
  Humans have needs motivating their behavior according to intensity and context. We also create preferences associated with each action's perceived pleasure, which is susceptible to changes over time. This makes decision-making more complex, requiring learning to balance needs and preferences according to the context.
  arXiv  Detail & Related papers  (2023-02-20T04:52:24Z)
• Pessimism meets VCG: Learning Dynamic Mechanism Design via Offline Reinforcement Learning [114.36124979578896]
  We design a dynamic mechanism using offline reinforcement learning algorithms. Our algorithm is based on the pessimism principle and only requires a mild assumption on the coverage of the offline data set.
  arXiv  Detail & Related papers  (2022-05-05T05:44:26Z)
• Learning Theory of Mind via Dynamic Traits Attribution [59.9781556714202]
  We propose a new neural ToM architecture that learns to generate a latent trait vector of an actor from the past trajectories. This trait vector then multiplicatively modulates the prediction mechanism via a fast weights' scheme in the prediction neural network. We empirically show that the fast weights provide a good inductive bias to model the character traits of agents and hence improves mindreading ability.
  arXiv  Detail & Related papers  (2022-04-17T11:21:18Z)
• Inference of Affordances and Active Motor Control in Simulated Agents [0.5161531917413706]
  We introduce an output-probabilistic, temporally predictive, modular artificial neural network architecture. We show that our architecture develops latent states that can be interpreted as affordance maps. In combination with active inference, we show that flexible, goal-directed behavior can be invoked.
  arXiv  Detail & Related papers  (2022-02-23T14:13:04Z)
• Information is Power: Intrinsic Control via Information Capture [110.3143711650806]
  We argue that a compact and general learning objective is to minimize the entropy of the agent's state visitation estimated using a latent state-space model. This objective induces an agent to both gather information about its environment, corresponding to reducing uncertainty, and to gain control over its environment, corresponding to reducing the unpredictability of future world states.
  arXiv  Detail & Related papers  (2021-12-07T18:50:42Z)
• What is Going on Inside Recurrent Meta Reinforcement Learning Agents? [63.58053355357644]
  Recurrent meta reinforcement learning (meta-RL) agents are agents that employ a recurrent neural network (RNN) for the purpose of "learning a learning algorithm" We shed light on the internal working mechanisms of these agents by reformulating the meta-RL problem using the Partially Observable Markov Decision Process (POMDP) framework.
  arXiv  Detail & Related papers  (2021-04-29T20:34:39Z)
• Continuous Learning and Adaptation with Membrane Potential and Activation Threshold Homeostasis [91.3755431537592]
  This paper presents the Membrane Potential and Activation Threshold Homeostasis (MPATH) neuron model. The model allows neurons to maintain a form of dynamic equilibrium by automatically regulating their activity when presented with input. Experiments demonstrate the model's ability to adapt to and continually learn from its input.
  arXiv  Detail & Related papers  (2021-04-22T04:01:32Z)
• Modulation of viability signals for self-regulatory control [1.370633147306388]
  We revisit the role of instrumental value as a driver of adaptive behavior. For reinforcement learning tasks, the distribution of preferences replaces the notion of reward.
  arXiv  Detail & Related papers  (2020-07-18T01:11:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.