Mutual Information State Intrinsic Control

• URL: http://arxiv.org/abs/2103.08107v1
• Date: Mon, 15 Mar 2021 03:03:36 GMT
• Title: Mutual Information State Intrinsic Control
• Authors: Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, Wei Xu
• Abstract summary: Intrinsically motivated RL attempts to remove this constraint by defining an intrinsic reward function. Motivated by the self-consciousness concept in psychology, we make a natural assumption that the agent knows what constitutes itself. We mathematically formalize this reward as the mutual information between the agent state and the surrounding state.
• Score: 91.38627985733068
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning has been shown to be highly successful at many challenging tasks. However, success heavily relies on well-shaped rewards. Intrinsically motivated RL attempts to remove this constraint by defining an intrinsic reward function. Motivated by the self-consciousness concept in psychology, we make a natural assumption that the agent knows what constitutes itself, and propose a new intrinsic objective that encourages the agent to have maximum control on the environment. We mathematically formalize this reward as the mutual information between the agent state and the surrounding state under the current agent policy. With this new intrinsic motivation, we are able to outperform previous methods, including being able to complete the pick-and-place task for the first time without using any task reward. A video showing experimental results is available at https://youtu.be/AUCwc9RThpk.

Related papers

• Multi Task Inverse Reinforcement Learning for Common Sense Reward [21.145179791929337]
  We show that inverse reinforcement learning, even when it succeeds in training an agent, does not learn a useful reward function. That is, training a new agent with the learned reward does not impair the desired behaviors. That is, multi-task inverse reinforcement learning can be applied to learn a useful reward function.
  arXiv  Detail & Related papers  (2024-02-17T19:49:00Z)
• Go Beyond Imagination: Maximizing Episodic Reachability with World Models [68.91647544080097]
  In this paper, we introduce a new intrinsic reward design called GoBI - Go Beyond Imagination. We apply learned world models to generate predicted future states with random actions. Our method greatly outperforms previous state-of-the-art methods on 12 of the most challenging Minigrid navigation tasks.
  arXiv  Detail & Related papers  (2023-08-25T20:30:20Z)
• Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments [0.0]
  An intrinsic reward function based on the principle of empowerment assigns rewards proportional to the amount of control the agent has over its own sensors. We implement a variation on a recently proposed intrinsically motivated agent, which we refer to as the 'curious' agent, and an empowerment-inspired agent. We compare the performance of both agents to that of an advantage actor-critic baseline in four sparse reward grid worlds.
  arXiv  Detail & Related papers  (2021-07-14T22:52:38Z)
• Explore and Control with Adversarial Surprise [78.41972292110967]
  Reinforcement learning (RL) provides a framework for learning goal-directed policies given user-specified rewards. We propose a new unsupervised RL technique based on an adversarial game which pits two policies against each other to compete over the amount of surprise an RL agent experiences. We show that our method leads to the emergence of complex skills by exhibiting clear phase transitions.
  arXiv  Detail & Related papers  (2021-07-12T17:58:40Z)
• Self-Supervised Exploration via Latent Bayesian Surprise [4.088019409160893]
  In this work, we propose a curiosity-based bonus as intrinsic reward for Reinforcement Learning. We extensively evaluate our model by measuring the agent's performance in terms of environment exploration. Our model is cheap and empirically shows state-of-the-art performance on several problems.
  arXiv  Detail & Related papers  (2021-04-15T14:40:16Z)
• Maximizing Information Gain in Partially Observable Environments via Prediction Reward [64.24528565312463]
  This paper tackles the challenge of using belief-based rewards for a deep RL agent. We derive the exact error between negative entropy and the expected prediction reward. This insight provides theoretical motivation for several fields using prediction rewards.
  arXiv  Detail & Related papers  (2020-05-11T08:13:49Z)
• Intrinsic Motivation for Encouraging Synergistic Behavior [55.10275467562764]
  We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks. Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own.
  arXiv  Detail & Related papers  (2020-02-12T19:34:51Z)
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning [102.05692309417047]
  In reinforcement learning, an agent learns to reach a set of goals by means of an external reward signal. In the natural world, intelligent organisms learn from internal drives, bypassing the need for external signals. We propose to formulate an intrinsic objective as the mutual information between the goal states and the controllable states.
  arXiv  Detail & Related papers  (2020-02-05T19:21:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.