Hierarchical Representation Learning for Markov Decision Processes

• URL: http://arxiv.org/abs/2106.01655v1
• Date: Thu, 3 Jun 2021 07:53:18 GMT
• Title: Hierarchical Representation Learning for Markov Decision Processes
• Authors: Lorenzo Steccanella, Simone Totaro, Anders Jonsson
• Abstract summary: We present a novel method for learning hierarchical representations of Markov decision processes. Our method works by partitioning the state space into subsets, and defines subtasks for performing transitions between the partitions. We empirically validate the method, by showing that it can successfully learn a useful hierarchical representation in a navigation domain.
• Score: 9.904746542801837
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In this paper we present a novel method for learning hierarchical representations of Markov decision processes. Our method works by partitioning the state space into subsets, and defines subtasks for performing transitions between the partitions. We formulate the problem of partitioning the state space as an optimization problem that can be solved using gradient descent given a set of sampled trajectories, making our method suitable for high-dimensional problems with large state spaces. We empirically validate the method, by showing that it can successfully learn a useful hierarchical representation in a navigation domain. Once learned, the hierarchical representation can be used to solve different tasks in the given domain, thus generalizing knowledge across tasks.

Related papers

• Hierarchical Average-Reward Linearly-solvable Markov Decision Processes [11.69049916139847]
  We introduce a novel approach to hierarchical reinforcement learning for Linearly-solvable Markov Decision Processes. Our approach allows learning low-level and high-level tasks simultaneously, without imposing limiting restrictions on the low-level tasks. Experiments show that our approach can outperform flat average-reward reinforcement learning by one or several orders of magnitude.
  arXiv  Detail & Related papers  (2024-07-09T09:06:44Z)
• Ideal Abstractions for Decision-Focused Learning [108.15241246054515]
  We propose a method that configures the output space automatically in order to minimize the loss of decision-relevant information. We demonstrate the method in two domains: data acquisition for deep neural network training and a closed-loop wildfire management task.
  arXiv  Detail & Related papers  (2023-03-29T23:31:32Z)
• Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation [88.14365009076907]
  Iterative refinement is a useful paradigm for representation learning. We develop an implicit differentiation approach that improves the stability and tractability of training.
  arXiv  Detail & Related papers  (2022-07-02T10:00:35Z)
• Supervised learning of sheared distributions using linearized optimal transport [64.53761005509386]
  In this paper we study supervised learning tasks on the space of probability measures. We approach this problem by embedding the space of probability measures into $L2$ spaces using the optimal transport framework. Regular machine learning techniques are used to achieve linear separability.
  arXiv  Detail & Related papers  (2022-01-25T19:19:59Z)
• Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems [91.3755431537592]
  We consider a control system of the form $dot x = sum_i=1lF_i(x)u_i$, with linear dependence in the controls. We use the corresponding flow to approximate the action of a diffeomorphism on a compact ensemble of points.
  arXiv  Detail & Related papers  (2021-10-24T08:57:46Z)
• Context-Specific Representation Abstraction for Deep Option Learning [43.68681795014662]
  We introduce Context-Specific Representation Abstraction for Deep Option Learning (CRADOL) CRADOL is a new framework that considers both temporal abstraction and context-specific representation abstraction to effectively reduce the size of the search over policy space. Specifically, our method learns a factored belief state representation that enables each option to learn a policy over only a subsection of the state space.
  arXiv  Detail & Related papers  (2021-09-20T22:50:01Z)
• Globally Optimal Hierarchical Reinforcement Learning for Linearly-Solvable Markov Decision Processes [0.0]
  We present a novel approach to hierarchical reinforcement learning for linearly-solvable Markov decision processes. We represent value functions on several levels of abstraction, and use the compositionality of subtasks to estimate the optimal values of the states in each partition.
  arXiv  Detail & Related papers  (2021-06-29T13:10:08Z)
• Machine learning the real discriminant locus [13.63199518246153]
  This article views locating the real discriminant locus as a supervised classification problem in machine learning. At each sample point, homotopy continuation is used to obtain the number of real solutions to the corresponding system. One application of having learned the real discriminant locus is to develop a real homotopy method that only tracks the real solution paths.
  arXiv  Detail & Related papers  (2020-06-24T22:18:08Z)
• There and Back Again: Revisiting Backpropagation Saliency Methods [87.40330595283969]
  Saliency methods seek to explain the predictions of a model by producing an importance map across each input sample. A popular class of such methods is based on backpropagating a signal and analyzing the resulting gradient. We propose a single framework under which several such methods can be unified.
  arXiv  Detail & Related papers  (2020-04-06T17:58:08Z)
• Spatial Pyramid Based Graph Reasoning for Semantic Segmentation [67.47159595239798]
  We apply graph convolution into the semantic segmentation task and propose an improved Laplacian. The graph reasoning is directly performed in the original feature space organized as a spatial pyramid. We achieve comparable performance with advantages in computational and memory overhead.
  arXiv  Detail & Related papers  (2020-03-23T12:28:07Z)
• Learning Discrete State Abstractions With Deep Variational Inference [7.273663549650618]
  We propose a method for learning approximate bisimulations, a type of state abstraction. We use a deep neural encoder to map states onto continuous embeddings. We map these embeddings onto a discrete representation using an action-conditioned hidden Markov model.
  arXiv  Detail & Related papers  (2020-03-09T17:58:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.