Mitigating Covariate Shift in Imitation Learning via Offline Data
Without Great Coverage
- URL: http://arxiv.org/abs/2106.03207v1
- Date: Sun, 6 Jun 2021 18:31:08 GMT
- Title: Mitigating Covariate Shift in Imitation Learning via Offline Data
Without Great Coverage
- Authors: Jonathan D. Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi,
Wen Sun
- Abstract summary: This paper studies offline Imitation Learning (IL) where an agent learns to imitate an expert demonstrator without additional online environment interactions.
Instead, the learner is presented with a static offline dataset of state-action-next state transition triples from a potentially less proficient behavior policy.
We introduce Model-based IL from Offline data (MILO) to solve the offline IL problem efficiently both in theory and in practice.
- Score: 27.122391441921664
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper studies offline Imitation Learning (IL) where an agent learns to
imitate an expert demonstrator without additional online environment
interactions. Instead, the learner is presented with a static offline dataset
of state-action-next state transition triples from a potentially less
proficient behavior policy. We introduce Model-based IL from Offline data
(MILO): an algorithmic framework that utilizes the static dataset to solve the
offline IL problem efficiently both in theory and in practice. In theory, even
if the behavior policy is highly sub-optimal compared to the expert, we show
that as long as the data from the behavior policy provides sufficient coverage
on the expert state-action traces (and with no necessity for a global coverage
over the entire state-action space), MILO can provably combat the covariate
shift issue in IL. Complementing our theory results, we also demonstrate that a
practical implementation of our approach mitigates covariate shift on benchmark
MuJoCo continuous control tasks. We demonstrate that with behavior policies
whose performances are less than half of that of the expert, MILO still
successfully imitates with an extremely low number of expert state-action pairs
while traditional offline IL method such as behavior cloning (BC) fails
completely. Source code is provided at https://github.com/jdchang1/milo.
Related papers
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning [26.53136644321385]
Imitation learning (IL) aims to mimic the behavior of an expert in a sequential decision making task by learning from demonstrations.
Online behavior cloning (BC) is thought to incur sample complexity with unfavorable quadratic dependence on the problem horizon.
We show that it is possible to achieve horizon-independent sample complexity in offline IL whenever the range of the cumulative payoffs is controlled.
arXiv Detail & Related papers (2024-07-20T23:31:56Z) - How to Leverage Diverse Demonstrations in Offline Imitation Learning [39.24627312800116]
Offline Imitation Learning (IL) with imperfect demonstrations has garnered increasing attention owing to the scarcity of expert data.
We introduce a simple yet effective data selection method that identifies positive behaviors based on their resultant states.
We then devise a lightweight behavior cloning algorithm capable of leveraging the expert and selected data correctly.
arXiv Detail & Related papers (2024-05-24T04:56:39Z) - Is Inverse Reinforcement Learning Harder than Standard Reinforcement
Learning? A Theoretical Perspective [55.36819597141271]
Inverse Reinforcement Learning (IRL) -- the problem of learning reward functions from demonstrations of an emphexpert policy -- plays a critical role in developing intelligent systems.
This paper provides the first line of efficient IRL in vanilla offline and online settings using samples and runtime.
As an application, we show that the learned rewards can emphtransfer to another target MDP with suitable guarantees.
arXiv Detail & Related papers (2023-11-29T00:09:01Z) - A Simple Solution for Offline Imitation from Observations and Examples
with Possibly Incomplete Trajectories [122.11358440078581]
offline imitation is useful in real-world scenarios where arbitrary interactions are costly and expert actions are unavailable.
We propose Trajectory-Aware Learning from Observations (TAILO) to solve MDPs where only task-specific expert states and task-agnostic non-expert state-action pairs are available.
arXiv Detail & Related papers (2023-11-02T15:41:09Z) - Offline Imitation Learning with Suboptimal Demonstrations via Relaxed
Distribution Matching [109.5084863685397]
offline imitation learning (IL) promises the ability to learn performant policies from pre-collected demonstrations without interactions with the environment.
We present RelaxDICE, which employs an asymmetrically-relaxed f-divergence for explicit support regularization.
Our method significantly outperforms the best prior offline method in six standard continuous control environments.
arXiv Detail & Related papers (2023-03-05T03:35:11Z) - Boosting Offline Reinforcement Learning via Data Rebalancing [104.3767045977716]
offline reinforcement learning (RL) is challenged by the distributional shift between learning policies and datasets.
We propose a simple yet effective method to boost offline RL algorithms based on the observation that resampling a dataset keeps the distribution support unchanged.
We dub our method ReD (Return-based Data Rebalance), which can be implemented with less than 10 lines of code change and adds negligible running time.
arXiv Detail & Related papers (2022-10-17T16:34:01Z) - Mutual Information Regularized Offline Reinforcement Learning [76.05299071490913]
We propose a novel MISA framework to approach offline RL from the perspective of Mutual Information between States and Actions in the dataset.
We show that optimizing this lower bound is equivalent to maximizing the likelihood of a one-step improved policy on the offline dataset.
We introduce 3 different variants of MISA, and empirically demonstrate that tighter mutual information lower bound gives better offline RL performance.
arXiv Detail & Related papers (2022-10-14T03:22:43Z) - Curriculum Offline Imitation Learning [72.1015201041391]
offline reinforcement learning tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment.
We propose textitCurriculum Offline Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return.
On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.
arXiv Detail & Related papers (2021-11-03T08:02:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.