Guaranteed Discovery of Controllable Latent States with Multi-Step Inverse Models

• URL: http://arxiv.org/abs/2207.08229v1
• Date: Sun, 17 Jul 2022 17:06:52 GMT
• Title: Guaranteed Discovery of Controllable Latent States with Multi-Step Inverse Models
• Authors: Alex Lamb, Riashat Islam, Yonathan Efroni, Aniket Didolkar, Dipendra Misra, Dylan Foster, Lekan Molu, Rajan Chari, Akshay Krishnamurthy, John Langford
• Abstract summary: Agent-Controllable State Discovery algorithm (AC-State) Algorithm consists of a multi-step inverse model (predicting actions from distant observations) with an information bottleneck. We demonstrate the discovery of controllable latent state in three domains: localizing a robot arm with distractions, exploring in a maze alongside other agents, and navigating in the Matterport house simulator.
• Score: 51.754160866582005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A person walking along a city street who tries to model all aspects of the world would quickly be overwhelmed by a multitude of shops, cars, and people moving in and out of view, following their own complex and inscrutable dynamics. Exploration and navigation in such an environment is an everyday task, requiring no vast exertion of mental resources. Is it possible to turn this fire hose of sensory information into a minimal latent state which is necessary and sufficient for an agent to successfully act in the world? We formulate this question concretely, and propose the Agent-Controllable State Discovery algorithm (AC-State), which has theoretical guarantees and is practically demonstrated to discover the \textit{minimal controllable latent state} which contains all of the information necessary for controlling the agent, while fully discarding all irrelevant information. This algorithm consists of a multi-step inverse model (predicting actions from distant observations) with an information bottleneck. AC-State enables localization, exploration, and navigation without reward or demonstrations. We demonstrate the discovery of controllable latent state in three domains: localizing a robot arm with distractions (e.g., changing lighting conditions and background), exploring in a maze alongside other agents, and navigating in the Matterport house simulator.

Related papers

• METRA: Scalable Unsupervised RL with Metric-Aware Abstraction [69.90741082762646]
  Metric-Aware Abstraction (METRA) is a novel unsupervised reinforcement learning objective. By learning to move in every direction in the latent space, METRA obtains a tractable set of diverse behaviors. We show that METRA can discover a variety of useful behaviors even in complex, pixel-based environments.
  arXiv  Detail & Related papers  (2023-10-13T06:43:11Z)
• Provably Sample-Efficient RL with Side Information about Latent Dynamics [12.461789905893026]
  We study reinforcement learning in settings where observations are high-dimensional, but where an RL agent has access to abstract knowledge about the structure of the state space. We present an algorithm, called TASID, that learns a robust policy in the target domain, with sample complexity that is in the horizon.
  arXiv  Detail & Related papers  (2022-05-27T21:07:03Z)
• Rapid Exploration for Open-World Navigation with Latent Goal Models [78.45339342966196]
  We describe a robotic learning system for autonomous exploration and navigation in diverse, open-world environments. At the core of our method is a learned latent variable model of distances and actions, along with a non-parametric topological memory of images. We use an information bottleneck to regularize the learned policy, giving us (i) a compact visual representation of goals, (ii) improved generalization capabilities, and (iii) a mechanism for sampling feasible goals for exploration.
  arXiv  Detail & Related papers  (2021-04-12T23:14:41Z)
• Guided Navigation from Multiple Viewpoints using Qualitative Spatial Reasoning [0.0]
  This work aims to develop algorithms capable of guiding a sensory deprived robot to a goal location. The main task considered in this work is, given a group of autonomous agents, the development and evaluation of algorithms capable of producing a set of high-level commands.
  arXiv  Detail & Related papers  (2020-11-03T00:34:26Z)
• Batch Exploration with Examples for Scalable Robotic Reinforcement Learning [63.552788688544254]
  Batch Exploration with Examples (BEE) explores relevant regions of the state-space guided by a modest number of human provided images of important states. BEE is able to tackle challenging vision-based manipulation tasks both in simulation and on a real Franka robot.
  arXiv  Detail & Related papers  (2020-10-22T17:49:25Z)
• Active Visual Information Gathering for Vision-Language Navigation [115.40768457718325]
  Vision-language navigation (VLN) is the task of entailing an agent to carry out navigational instructions inside photo-realistic environments. One of the key challenges in VLN is how to conduct a robust navigation by mitigating the uncertainty caused by ambiguous instructions and insufficient observation of the environment. This work draws inspiration from human navigation behavior and endows an agent with an active information gathering ability for a more intelligent VLN policy.
  arXiv  Detail & Related papers  (2020-07-15T23:54:20Z)
• Asynchronous Multi Agent Active Search [6.587280549237275]
  We propose two distinct active search algorithms called SPATS (Sparse Parallel Asynchronous Thompson Sampling) and LATSI (LAplace Thompson Sampling with Information gain) We consider that targets are sparsely located around the environment in keeping with compressive sensing assumptions. We provide simulation results as well as theoretical analysis to demonstrate the efficacy of our proposed algorithms.
  arXiv  Detail & Related papers  (2020-06-25T22:17:20Z)
• Learning to Move with Affordance Maps [57.198806691838364]
  The ability to autonomously explore and navigate a physical space is a fundamental requirement for virtually any mobile autonomous agent. Traditional SLAM-based approaches for exploration and navigation largely focus on leveraging scene geometry. We show that learned affordance maps can be used to augment traditional approaches for both exploration and navigation, providing significant improvements in performance.
  arXiv  Detail & Related papers  (2020-01-08T04:05:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.