Observation Space Matters: Benchmark and Optimization Algorithm
- URL: http://arxiv.org/abs/2011.00756v1
- Date: Mon, 2 Nov 2020 05:40:31 GMT
- Title: Observation Space Matters: Benchmark and Optimization Algorithm
- Authors: Joanne Taery Kim and Sehoon Ha
- Abstract summary: We propose a search algorithm to find the optimal observation spaces.
We demonstrate that our algorithm significantly improves learning speed compared to manually designed observation spaces.
- Score: 20.503293998529024
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent advances in deep reinforcement learning (deep RL) enable researchers
to solve challenging control problems, from simulated environments to
real-world robotic tasks. However, deep RL algorithms are known to be sensitive
to the problem formulation, including observation spaces, action spaces, and
reward functions. There exist numerous choices for observation spaces but they
are often designed solely based on prior knowledge due to the lack of
established principles. In this work, we conduct benchmark experiments to
verify common design choices for observation spaces, such as Cartesian
transformation, binary contact flags, a short history, or global positions.
Then we propose a search algorithm to find the optimal observation spaces,
which examines various candidate observation spaces and removes unnecessary
observation channels with a Dropout-Permutation test. We demonstrate that our
algorithm significantly improves learning speed compared to manually designed
observation spaces. We also analyze the proposed algorithm by evaluating
different hyperparameters.
Related papers
- Earth Observation Satellite Scheduling with Graph Neural Networks [1.1684839631276702]
This paper presents a new technique for selecting and scheduling observations based on Graph Neural Networks (GNNs) and Deep Reinforcement Learning (DRL)
Our simulations show that it is able to learn on small problem instances and generalize to larger real-world instances, with very competitive performance compared to traditional approaches.
arXiv Detail & Related papers (2024-08-27T13:10:26Z) - Gradient-Informed Quality Diversity for the Illumination of Discrete
Spaces [7.799824794686343]
Quality Diversity (QD) algorithms have been proposed to search for a large collection of both diverse and high-performing solutions instead of a single set of local optima.
We introduce a Gradient-Informed Discrete Emitter (ME-GIDE), which extends QD with differentiable functions over discrete search spaces.
We evaluate our method on challenging benchmarks including protein design and discrete latent space illumination and find that our method outperforms state-of-the-art QD algorithms in all benchmarks.
arXiv Detail & Related papers (2023-06-08T12:04:52Z) - Representation Learning with Multi-Step Inverse Kinematics: An Efficient
and Optimal Approach to Rich-Observation RL [106.82295532402335]
Existing reinforcement learning algorithms suffer from computational intractability, strong statistical assumptions, and suboptimal sample complexity.
We provide the first computationally efficient algorithm that attains rate-optimal sample complexity with respect to the desired accuracy level.
Our algorithm, MusIK, combines systematic exploration with representation learning based on multi-step inverse kinematics.
arXiv Detail & Related papers (2023-04-12T14:51:47Z) - Towards Spatial Equilibrium Object Detection [88.9747319572368]
In this paper, we study the spatial disequilibrium problem of modern object detectors.
We propose to quantify this problem by measuring the detection performance over zones.
This motivates us to design a more generalized measurement, termed Spatial equilibrium Precision.
arXiv Detail & Related papers (2023-01-14T17:33:26Z) - Memetic algorithms for Spatial Partitioning problems [26.73720392872553]
In this article, we focus on a specific type of SOP called spatial partitioning on real-world datasets.
We put forward a simple yet effective algorithm called swarm-based spatial memetic algorithm (SPATIAL) and test it on the school (re)districting problem.
arXiv Detail & Related papers (2022-08-04T20:05:46Z) - Active Exploration via Experiment Design in Markov Chains [86.41407938210193]
A key challenge in science and engineering is to design experiments to learn about some unknown quantity of interest.
We propose an algorithm that efficiently selects policies whose measurement allocation converges to the optimal one.
In addition to our theoretical analysis, we showcase our framework on applications in ecological surveillance and pharmacology.
arXiv Detail & Related papers (2022-06-29T00:04:40Z) - k-Means Maximum Entropy Exploration [55.81894038654918]
Exploration in continuous spaces with sparse rewards is an open problem in reinforcement learning.
We introduce an artificial curiosity algorithm based on lower bounding an approximation to the entropy of the state visitation distribution.
We show that our approach is both computationally efficient and competitive on benchmarks for exploration in high-dimensional, continuous spaces.
arXiv Detail & Related papers (2022-05-31T09:05:58Z) - Transfer RL across Observation Feature Spaces via Model-Based
Regularization [9.660642248872973]
In many reinforcement learning (RL) applications, the observation space is specified by human developers and restricted by physical realizations.
We propose a novel algorithm which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task.
Our algorithm works for drastic changes of observation space without any inter-task mapping or any prior knowledge of the target task.
arXiv Detail & Related papers (2022-01-01T22:41:19Z) - Exploring Complicated Search Spaces with Interleaving-Free Sampling [127.07551427957362]
In this paper, we build the search algorithm upon a complicated search space with long-distance connections.
We present a simple yet effective algorithm named textbfIF-NAS, where we perform a periodic sampling strategy to construct different sub-networks.
In the proposed search space, IF-NAS outperform both random sampling and previous weight-sharing search algorithms by a significant margin.
arXiv Detail & Related papers (2021-12-05T06:42:48Z) - AutoSpace: Neural Architecture Search with Less Human Interference [84.42680793945007]
Current neural architecture search (NAS) algorithms still require expert knowledge and effort to design a search space for network construction.
We propose a novel differentiable evolutionary framework named AutoSpace, which evolves the search space to an optimal one.
With the learned search space, the performance of recent NAS algorithms can be improved significantly compared with using previously manually designed spaces.
arXiv Detail & Related papers (2021-03-22T13:28:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.