Ego-Foresight: Agent Visuomotor Prediction as Regularization for RL

• URL: http://arxiv.org/abs/2407.01570v1
• Date: Mon, 27 May 2024 13:32:43 GMT
• Title: Ego-Foresight: Agent Visuomotor Prediction as Regularization for RL
• Authors: Manuel S. Nunes, Atabak Dehban, Yiannis Demiris, José Santos-Victor,
• Abstract summary: Ego-Foresight is a self-supervised method for disentangling agent and environment based on motion and prediction. We show that visuomotor prediction of the agent provides regularization to the RL algorithm, by encouraging the actions to stay within predictable bounds. We integrate Ego-Foresight with a model-free RL algorithm to solve simulated robotic manipulation tasks, showing an average improvement of 23% in efficiency and 8% in performance.
• Score: 34.6883445484835
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the significant advancements in Deep Reinforcement Learning (RL) observed in the last decade, the amount of training experience necessary to learn effective policies remains one of the primary concerns both in simulated and real environments. Looking to solve this issue, previous work has shown that improved training efficiency can be achieved by separately modeling agent and environment, but usually requiring a supervisory agent mask. In contrast to RL, humans can perfect a new skill from a very small number of trials and in most cases do so without a supervisory signal, making neuroscientific studies of human development a valuable source of inspiration for RL. In particular, we explore the idea of motor prediction, which states that humans develop an internal model of themselves and of the consequences that their motor commands have on the immediate sensory inputs. Our insight is that the movement of the agent provides a cue that allows the duality between agent and environment to be learned. To instantiate this idea, we present Ego-Foresight, a self-supervised method for disentangling agent and environment based on motion and prediction. Our main finding is that visuomotor prediction of the agent provides regularization to the RL algorithm, by encouraging the actions to stay within predictable bounds. To test our approach, we first study the ability of our model to visually predict agent movement irrespective of the environment, in real-world robotic interactions. Then, we integrate Ego-Foresight with a model-free RL algorithm to solve simulated robotic manipulation tasks, showing an average improvement of 23% in efficiency and 8% in performance.

Related papers

• Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance [95.03771007780976]
  We tackle the challenge of developing proactive agents capable of anticipating and initiating tasks without explicit human instructions. First, we collect real-world human activities to generate proactive task predictions. These predictions are labeled by human annotators as either accepted or rejected. The labeled data is used to train a reward model that simulates human judgment.
  arXiv  Detail & Related papers  (2024-10-16T08:24:09Z)
• In-context Learning for Automated Driving Scenarios [15.325910109153616]
  One of the key challenges in current Reinforcement Learning (RL)-based Automated Driving (AD) agents is achieving flexible, precise, and human-like behavior cost-effectively. This paper introduces an innovative approach utilizing Large Language Models (LLMs) to intuitively and effectively optimize RL reward functions in a human-centric way.
  arXiv  Detail & Related papers  (2024-05-07T09:04:52Z)
• HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Driving [2.807187711407621]
  We propose an enhanced human-in-the-loop reinforcement learning method, termed the Human as AI mentor-based deep reinforcement learning (HAIM-DRL) framework. We first introduce an innovative learning paradigm that effectively injects human intelligence into AI, termed Human as AI mentor (HAIM) In this paradigm, the human expert serves as a mentor to the AI agent, while the agent could be guided to minimize traffic flow disturbance.
  arXiv  Detail & Related papers  (2024-01-06T08:30:14Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)
• Learning Theory of Mind via Dynamic Traits Attribution [59.9781556714202]
  We propose a new neural ToM architecture that learns to generate a latent trait vector of an actor from the past trajectories. This trait vector then multiplicatively modulates the prediction mechanism via a fast weights' scheme in the prediction neural network. We empirically show that the fast weights provide a good inductive bias to model the character traits of agents and hence improves mindreading ability.
  arXiv  Detail & Related papers  (2022-04-17T11:21:18Z)
• Modeling Bounded Rationality in Multi-Agent Simulations Using Rationally Inattentive Reinforcement Learning [85.86440477005523]
  We study more human-like RL agents which incorporate an established model of human-irrationality, the Rational Inattention (RI) model. RIRL models the cost of cognitive information processing using mutual information. We show that using RIRL yields a rich spectrum of new equilibrium behaviors that differ from those found under rational assumptions.
  arXiv  Detail & Related papers  (2022-01-18T20:54:00Z)
• Assessing Human Interaction in Virtual Reality With Continually Learning Prediction Agents Based on Reinforcement Learning Algorithms: A Pilot Study [6.076137037890219]
  We investigate how the interaction between a human and a continually learning prediction agent develops as the agent develops competency. We develop a virtual reality environment and a time-based prediction task wherein learned predictions from a reinforcement learning (RL) algorithm augment human predictions. Our findings suggest that human trust of the system may be influenced by early interactions with the agent, and that trust in turn affects strategic behaviour.
  arXiv  Detail & Related papers  (2021-12-14T22:46:44Z)
• What is Going on Inside Recurrent Meta Reinforcement Learning Agents? [63.58053355357644]
  Recurrent meta reinforcement learning (meta-RL) agents are agents that employ a recurrent neural network (RNN) for the purpose of "learning a learning algorithm" We shed light on the internal working mechanisms of these agents by reformulating the meta-RL problem using the Partially Observable Markov Decision Process (POMDP) framework.
  arXiv  Detail & Related papers  (2021-04-29T20:34:39Z)
• Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedback [10.138798960466222]
  Reinforcement Learning (RL) agents with human feedback can dramatically improve various aspects of learning. Previous methods require human observer to give inputs explicitly, burdening the human in the loop of RL agent's learning process. We investigate capturing human's intrinsic reactions as implicit (and natural) feedback through EEG in the form of error-related potentials (ErrP)
  arXiv  Detail & Related papers  (2020-06-30T03:13:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.