Visual processing in context of reinforcement learning
- URL: http://arxiv.org/abs/2208.12525v1
- Date: Fri, 26 Aug 2022 09:30:51 GMT
- Title: Visual processing in context of reinforcement learning
- Authors: Hlynur Dav\'i{\dh} Hlynsson
- Abstract summary: This thesis introduces three different representation learning algorithms that have access to different subsets of the data sources that traditional RL algorithms use.
We conclude that including unsupervised representation learning in RL problem-solving pipelines can speed up learning.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Although deep reinforcement learning (RL) has recently enjoyed many
successes, its methods are still data inefficient, which makes solving numerous
problems prohibitively expensive in terms of data. We aim to remedy this by
taking advantage of the rich supervisory signal in unlabeled data for learning
state representations. This thesis introduces three different representation
learning algorithms that have access to different subsets of the data sources
that traditional RL algorithms use:
(i) GRICA is inspired by independent component analysis (ICA) and trains a
deep neural network to output statistically independent features of the input.
GrICA does so by minimizing the mutual information between each feature and the
other features. Additionally, GrICA only requires an unsorted collection of
environment states.
(ii) Latent Representation Prediction (LARP) requires more context: in
addition to requiring a state as an input, it also needs the previous state and
an action that connects them. This method learns state representations by
predicting the representation of the environment's next state given a current
state and action. The predictor is used with a graph search algorithm.
(iii) RewPred learns a state representation by training a deep neural network
to learn a smoothed version of the reward function. The representation is used
for preprocessing inputs to deep RL, while the reward predictor is used for
reward shaping. This method needs only state-reward pairs from the environment
for learning the representation.
We discover that every method has their strengths and weaknesses, and
conclude from our experiments that including unsupervised representation
learning in RL problem-solving pipelines can speed up learning.
Related papers
- Towards Principled Representation Learning from Videos for Reinforcement Learning [23.877731515619868]
We study pre-training representations for decision-making using video data.
We focus on learning the latent state representations of the underlying MDP using video data.
arXiv Detail & Related papers (2024-03-20T17:28:17Z) - State Sequences Prediction via Fourier Transform for Representation
Learning [111.82376793413746]
We propose State Sequences Prediction via Fourier Transform (SPF), a novel method for learning expressive representations efficiently.
We theoretically analyze the existence of structural information in state sequences, which is closely related to policy performance and signal regularity.
Experiments demonstrate that the proposed method outperforms several state-of-the-art algorithms in terms of both sample efficiency and performance.
arXiv Detail & Related papers (2023-10-24T14:47:02Z) - A Study of Forward-Forward Algorithm for Self-Supervised Learning [65.268245109828]
We study the performance of forward-forward vs. backpropagation for self-supervised representation learning.
Our main finding is that while the forward-forward algorithm performs comparably to backpropagation during (self-supervised) training, the transfer performance is significantly lagging behind in all the studied settings.
arXiv Detail & Related papers (2023-09-21T10:14:53Z) - Speech representation learning: Learning bidirectional encoders with
single-view, multi-view, and multi-task methods [7.1345443932276424]
This thesis focuses on representation learning for sequence data over time or space.
It aims to improve downstream sequence prediction tasks by using the learned representations.
arXiv Detail & Related papers (2023-07-25T20:38:55Z) - Value-Consistent Representation Learning for Data-Efficient
Reinforcement Learning [105.70602423944148]
We propose a novel method, called value-consistent representation learning (VCR), to learn representations that are directly related to decision-making.
Instead of aligning this imagined state with a real state returned by the environment, VCR applies a $Q$-value head on both states and obtains two distributions of action values.
It has been demonstrated that our methods achieve new state-of-the-art performance for search-free RL algorithms.
arXiv Detail & Related papers (2022-06-25T03:02:25Z) - Exploratory State Representation Learning [63.942632088208505]
We propose a new approach called XSRL (eXploratory State Representation Learning) to solve the problems of exploration and SRL in parallel.
On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations.
On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a $k$-step learning progress bonus to form the objective of a discovery policy.
arXiv Detail & Related papers (2021-09-28T10:11:07Z) - Reasoning-Modulated Representations [85.08205744191078]
We study a common setting where our task is not purely opaque.
Our approach paves the way for a new class of data-efficient representation learning.
arXiv Detail & Related papers (2021-07-19T13:57:13Z) - Predicting What You Already Know Helps: Provable Self-Supervised
Learning [60.27658820909876]
Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data.
We show a mechanism exploiting the statistical connections between certain em reconstruction-based pretext tasks that guarantee to learn a good representation.
We prove the linear layer yields small approximation error even for complex ground truth function class.
arXiv Detail & Related papers (2020-08-03T17:56:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.