CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
- URL: http://arxiv.org/abs/2110.02102v1
- Date: Tue, 5 Oct 2021 15:04:01 GMT
- Title: CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning
- Authors: Carolin Benjamins, Theresa Eimer, Frederik Schubert, Andr\'e
Biedenkapp, Bodo Rosenhahn, Frank Hutter, Marius Lindauer
- Abstract summary: We present CARL, a collection of well-known RL environments extended to contextual RL problems.
We provide first evidence that disentangling representation learning of the states from the policy learning with the context facilitates better generalization.
- Score: 45.52724876199729
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While Reinforcement Learning has made great strides towards solving ever more
complicated tasks, many algorithms are still brittle to even slight changes in
their environment. This is a limiting factor for real-world applications of RL.
Although the research community continuously aims at improving both robustness
and generalization of RL algorithms, unfortunately it still lacks an
open-source set of well-defined benchmark problems based on a consistent
theoretical framework, which allows comparing different approaches in a fair,
reliable and reproducibleway. To fill this gap, we propose CARL, a collection
of well-known RL environments extended to contextual RL problems to study
generalization. We show the urgent need of such benchmarks by demonstrating
that even simple toy environments become challenging for commonly used
approaches if different contextual instances of this task have to be
considered. Furthermore, CARL allows us to provide first evidence that
disentangling representation learning of the states from the policy learning
with the context facilitates better generalization. By providing variations of
diverse benchmarks from classic control, physical simulations, games and a
real-world application of RNA design, CARL will allow the community to derive
many more such insights on a solid empirical foundation.
Related papers
- Multi-Agent Environments for Vehicle Routing Problems [1.0179489519625304]
We propose a library composed of multi-agent environments that simulates classic vehicle routing problems.
The library, built on PyTorch, provides a flexible modular architecture design that allows easy customization and incorporation of new routing problems.
arXiv Detail & Related papers (2024-11-21T18:46:23Z) - Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review [50.67937325077047]
This paper is devoted to a comprehensive review of realizing the sample efficiency and generalization of RL algorithms through transfer and inverse reinforcement learning (T-IRL)
Our findings denote that a majority of recent research works have dealt with the aforementioned challenges by utilizing human-in-the-loop and sim-to-real strategies.
Under the IRL structure, training schemes that require a low number of experience transitions and extension of such frameworks to multi-agent and multi-intention problems have been the priority of researchers in recent years.
arXiv Detail & Related papers (2024-11-15T15:18:57Z) - Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach [2.3020018305241337]
This paper is the first to propose considering the RRL problems within the positional differential game theory.
Namely, we prove that under Isaacs's condition, the same Q-function can be utilized as an approximate solution of both minimax and maximin Bellman equations.
We present the Isaacs Deep Q-Network algorithms and demonstrate their superiority compared to other baseline RRL and Multi-Agent RL algorithms in various environments.
arXiv Detail & Related papers (2024-05-03T12:21:43Z) - Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning [48.79569442193824]
We show that COMRL algorithms are essentially optimizing the same mutual information objective between the task variable $M$ and its latent representation $Z$ by implementing various approximate bounds.
This work lays the information theoretic foundation for COMRL methods, leading to a better understanding of task representation learning in the context of reinforcement learning.
arXiv Detail & Related papers (2024-02-04T09:58:42Z) - Blending Imitation and Reinforcement Learning for Robust Policy
Improvement [16.588397203235296]
Imitation learning (IL) utilizes oracles to improve sample efficiency.
RPI draws on the strengths of IL, using oracle queries to facilitate exploration.
RPI is capable of learning from and improving upon a diverse set of black-box oracles.
arXiv Detail & Related papers (2023-10-03T01:55:54Z) - ContainerGym: A Real-World Reinforcement Learning Benchmark for Resource
Allocation [1.6058099298620425]
ContainerGym is a benchmark for reinforcement learning inspired by a real-world industrial resource allocation task.
The proposed benchmark encodes challenges commonly encountered in real-world sequential decision making problems.
It can be configured to instantiate problems of varying degrees of difficulty.
arXiv Detail & Related papers (2023-07-06T13:44:29Z) - MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion
Control in Real Networks [63.24965775030673]
We propose a novel Reinforcement Learning (RL) approach to design generic Congestion Control (CC) algorithms.
Our solution, MARLIN, uses the Soft Actor-Critic algorithm to maximize both entropy and return.
We trained MARLIN on a real network with varying background traffic patterns to overcome the sim-to-real mismatch.
arXiv Detail & Related papers (2023-02-02T18:27:20Z) - Contextualize Me -- The Case for Context in Reinforcement Learning [49.794253971446416]
Contextual Reinforcement Learning (cRL) provides a framework to model such changes in a principled manner.
We show how cRL contributes to improving zero-shot generalization in RL through meaningful benchmarks and structured reasoning about generalization tasks.
arXiv Detail & Related papers (2022-02-09T15:01:59Z) - Autonomous Reinforcement Learning: Formalism and Benchmarking [106.25788536376007]
Real-world embodied learning, such as that performed by humans and animals, is situated in a continual, non-episodic world.
Common benchmark tasks in RL are episodic, with the environment resetting between trials to provide the agent with multiple attempts.
This discrepancy presents a major challenge when attempting to take RL algorithms developed for episodic simulated environments and run them on real-world platforms.
arXiv Detail & Related papers (2021-12-17T16:28:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.