Locally Constrained Representations in Reinforcement Learning
- URL: http://arxiv.org/abs/2209.09441v2
- Date: Fri, 9 Feb 2024 01:16:15 GMT
- Title: Locally Constrained Representations in Reinforcement Learning
- Authors: Somjit Nath, Rushiv Arora and Samira Ebrahimi Kahou
- Abstract summary: The success of Reinforcement Learning heavily relies on the ability to learn robust representations from the observations of the environment.
In most cases, the representations learned purely by the reinforcement learning loss can differ vastly across states depending on how the value functions change.
We propose locally constrained representations, where an auxiliary loss forces the state representations to be predictable by the representations of the neighboring states.
- Score: 5.865719902445064
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The success of Reinforcement Learning (RL) heavily relies on the ability to
learn robust representations from the observations of the environment. In most
cases, the representations learned purely by the reinforcement learning loss
can differ vastly across states depending on how the value functions change.
However, the representations learned need not be very specific to the task at
hand. Relying only on the RL objective may yield representations that vary
greatly across successive time steps. In addition, since the RL loss has a
changing target, the representations learned would depend on how good the
current values/policies are. Thus, disentangling the representations from the
main task would allow them to focus not only on the task-specific features but
also the environment dynamics. To this end, we propose locally constrained
representations, where an auxiliary loss forces the state representations to be
predictable by the representations of the neighboring states. This encourages
the representations to be driven not only by the value/policy learning but also
by an additional loss that constrains the representations from over-fitting to
the value loss. We evaluate the proposed method on several known benchmarks and
observe strong performance. Especially in continuous control tasks, our
experiments show a significant performance improvement.
Related papers
- Learning Interpretable Policies in Hindsight-Observable POMDPs through
Partially Supervised Reinforcement Learning [57.67629402360924]
We introduce the Partially Supervised Reinforcement Learning (PSRL) framework.
At the heart of PSRL is the fusion of both supervised and unsupervised learning.
We show that PSRL offers a potent balance, enhancing model interpretability while preserving, and often significantly outperforming, the performance benchmarks set by traditional methods.
arXiv Detail & Related papers (2024-02-14T16:23:23Z) - Harnessing Discrete Representations For Continual Reinforcement Learning [8.61539229796467]
We investigate the advantages of representing observations as vectors of categorical values within the context of reinforcement learning.
We find that, when compared to traditional continuous representations, world models learned over discrete representations accurately model more of the world with less capacity.
arXiv Detail & Related papers (2023-12-02T18:55:26Z) - Leveraging sparse and shared feature activations for disentangled
representation learning [112.22699167017471]
We propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation.
We validate our approach on six real world distribution shift benchmarks, and different data modalities.
arXiv Detail & Related papers (2023-04-17T01:33:24Z) - Representation Learning in Deep RL via Discrete Information Bottleneck [39.375822469572434]
We study how information bottlenecks can be used to construct latent states efficiently in the presence of task-irrelevant information.
We propose architectures that utilize variational and discrete information bottlenecks, coined as RepDIB, to learn structured factorized representations.
arXiv Detail & Related papers (2022-12-28T14:38:12Z) - Neural Distillation as a State Representation Bottleneck in
Reinforcement Learning [4.129225533930966]
We argue that distillation can be used to learn a state representation displaying favorable characteristics.
We first evaluate these criteria and verify the contribution of distillation on state representation on a toy environment based on the standard inverted pendulum problem.
arXiv Detail & Related papers (2022-10-05T13:00:39Z) - Exploratory State Representation Learning [63.942632088208505]
We propose a new approach called XSRL (eXploratory State Representation Learning) to solve the problems of exploration and SRL in parallel.
On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations.
On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a $k$-step learning progress bonus to form the objective of a discovery policy.
arXiv Detail & Related papers (2021-09-28T10:11:07Z) - Odd-One-Out Representation Learning [1.6822770693792826]
We show that a weakly-supervised downstream task based on odd-one-out observations is suitable for model selection.
We also show that a bespoke metric-learning VAE model which performs highly on this task also out-performs other standard unsupervised and a weakly-supervised disentanglement model.
arXiv Detail & Related papers (2020-12-14T22:01:15Z) - Towards Learning Controllable Representations of Physical Systems [9.088303226909279]
Learned representations of dynamical systems reduce dimensionality, potentially supporting downstream reinforcement learning (RL)
We consider the relationship between the true state and the corresponding representations, proposing that ideally each representation corresponds to a unique state.
These metrics are shown to predict reinforcement learning performance in a simulated peg-in-hole task when comparing variants of autoencoder-based representations.
arXiv Detail & Related papers (2020-11-16T17:15:57Z) - Learning Invariant Representations for Reinforcement Learning without
Reconstruction [98.33235415273562]
We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction.
Bisimulation metrics quantify behavioral similarity between states in continuous MDPs.
We demonstrate the effectiveness of our method at disregarding task-irrelevant information using modified visual MuJoCo tasks.
arXiv Detail & Related papers (2020-06-18T17:59:35Z) - Fairness by Learning Orthogonal Disentangled Representations [50.82638766862974]
We propose a novel disentanglement approach to invariant representation problem.
We enforce the meaningful representation to be agnostic to sensitive information by entropy.
The proposed approach is evaluated on five publicly available datasets.
arXiv Detail & Related papers (2020-03-12T11:09:15Z) - Weakly-Supervised Disentanglement Without Compromises [53.55580957483103]
Intelligent agents should be able to learn useful representations by observing changes in their environment.
We model such observations as pairs of non-i.i.d. images sharing at least one of the underlying factors of variation.
We show that only knowing how many factors have changed, but not which ones, is sufficient to learn disentangled representations.
arXiv Detail & Related papers (2020-02-07T16:39:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.