Predictive Coding for Locally-Linear Control
- URL: http://arxiv.org/abs/2003.01086v1
- Date: Mon, 2 Mar 2020 18:20:41 GMT
- Title: Predictive Coding for Locally-Linear Control
- Authors: Rui Shu, Tung Nguyen, Yinlam Chow, Tuan Pham, Khoat Than, Mohammad
Ghavamzadeh, Stefano Ermon, Hung H. Bui
- Abstract summary: High-dimensional observations and unknown dynamics are major challenges when applying optimal control to many real-world decision making tasks.
The Learning Controllable Embedding (LCE) framework addresses these challenges by embedding the observations into a lower dimensional latent space.
We show theoretically that explicit next-observation prediction can be replaced with predictive coding.
- Score: 92.35650774524399
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: High-dimensional observations and unknown dynamics are major challenges when
applying optimal control to many real-world decision making tasks. The Learning
Controllable Embedding (LCE) framework addresses these challenges by embedding
the observations into a lower dimensional latent space, estimating the latent
dynamics, and then performing control directly in the latent space. To ensure
the learned latent dynamics are predictive of next-observations, all existing
LCE approaches decode back into the observation space and explicitly perform
next-observation prediction---a challenging high-dimensional task that
furthermore introduces a large number of nuisance parameters (i.e., the
decoder) which are discarded during control. In this paper, we propose a novel
information-theoretic LCE approach and show theoretically that explicit
next-observation prediction can be replaced with predictive coding. We then use
predictive coding to develop a decoder-free LCE model whose latent dynamics are
amenable to locally-linear control. Extensive experiments on benchmark tasks
show that our model reliably learns a controllable latent space that leads to
superior performance when compared with state-of-the-art LCE baselines.
Related papers
- Trajectory Forecasting through Low-Rank Adaptation of Discrete Latent Codes [36.12653178844828]
Trajectory forecasting is crucial for video surveillance analytics, as it enables the anticipation of future movements for a set of agents.
We introduce Vector Quantized Variational Autoencoders (VQ-VAEs), which utilize a discrete latent space to tackle the issue of posterior collapse.
We show that such a two-fold framework, augmented with instance-level discretization, leads to accurate and diverse forecasts.
arXiv Detail & Related papers (2024-05-31T10:13:17Z) - Rich-Observation Reinforcement Learning with Continuous Latent Dynamics [43.84391209459658]
We introduce a new theoretical framework, RichCLD (Rich-Observation RL with Continuous Latent Dynamics), in which the agent performs control based on high-dimensional observations.
Our main contribution is a new algorithm for this setting that is provably statistically and computationally efficient.
arXiv Detail & Related papers (2024-05-29T17:02:49Z) - Controllability-Constrained Deep Network Models for Enhanced Control of
Dynamical Systems [4.948174943314265]
Control of a dynamical system without the knowledge of dynamics is an important and challenging task.
Modern machine learning approaches, such as deep neural networks (DNNs), allow for the estimation of a dynamics model from control inputs and corresponding state observation outputs.
We propose a control-theoretical method that explicitly enhances models estimated from data with controllability.
arXiv Detail & Related papers (2023-11-11T00:04:26Z) - Predictive Experience Replay for Continual Visual Control and
Forecasting [62.06183102362871]
We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting.
We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting.
Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
arXiv Detail & Related papers (2023-03-12T05:08:03Z) - Can Direct Latent Model Learning Solve Linear Quadratic Gaussian
Control? [75.14973944905216]
We study the task of learning state representations from potentially high-dimensional observations.
We pursue a direct latent model learning approach, where a dynamic model in some latent state space is learned by predicting quantities directly related to planning.
arXiv Detail & Related papers (2022-12-30T01:42:04Z) - Learning Self-Modulating Attention in Continuous Time Space with
Applications to Sequential Recommendation [102.24108167002252]
We propose a novel attention network, named self-modulating attention, that models the complex and non-linearly evolving dynamic user preferences.
We empirically demonstrate the effectiveness of our method on top-N sequential recommendation tasks, and the results on three large-scale real-world datasets show that our model can achieve state-of-the-art performance.
arXiv Detail & Related papers (2022-03-30T03:54:11Z) - Temporal Predictive Coding For Model-Based Planning In Latent Space [80.99554006174093]
We present an information-theoretic approach that employs temporal predictive coding to encode elements in the environment that can be predicted across time.
We evaluate our model on a challenging modification of standard DMControl tasks where the background is replaced with natural videos that contain complex but irrelevant information to the planning task.
arXiv Detail & Related papers (2021-06-14T04:31:15Z) - Trajectory-wise Multiple Choice Learning for Dynamics Generalization in
Reinforcement Learning [137.39196753245105]
We present a new model-based reinforcement learning algorithm that learns a multi-headed dynamics model for dynamics generalization.
We incorporate context learning, which encodes dynamics-specific information from past experiences into the context latent vector.
Our method exhibits superior zero-shot generalization performance across a variety of control tasks, compared to state-of-the-art RL methods.
arXiv Detail & Related papers (2020-10-26T03:20:42Z) - NewtonianVAE: Proportional Control and Goal Identification from Pixels
via Physical Latent Spaces [9.711378389037812]
We introduce a latent dynamics learning framework that is uniquely designed to induce proportional controlability in the latent space.
We show that our learned dynamics model enables proportional control from pixels, dramatically simplifies and accelerates behavioural cloning of vision-based controllers, and provides interpretable goal discovery when applied to imitation learning of switching controllers from demonstration.
arXiv Detail & Related papers (2020-06-02T21:41:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.