Deconfounded Imitation Learning
- URL: http://arxiv.org/abs/2211.02667v1
- Date: Fri, 4 Nov 2022 18:00:02 GMT
- Title: Deconfounded Imitation Learning
- Authors: Risto Vuorio, Johann Brehmer, Hanno Ackermann, Daniel Dijkman, Taco
Cohen, Pim de Haan
- Abstract summary: We introduce an algorithm for deconfounded imitation learning, which trains an inference model jointly with a latent-conditional policy.
We show in theory and practice that this algorithm converges to the correct interventional imitation policy, and can under certain assumptions achieve anally optimal imitation performance.
- Score: 19.0922018199264
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Standard imitation learning can fail when the expert demonstrators have
different sensory inputs than the imitating agent. This is because partial
observability gives rise to hidden confounders in the causal graph. We break
down the space of confounded imitation learning problems and identify three
settings with different data requirements in which the correct imitation policy
can be identified. We then introduce an algorithm for deconfounded imitation
learning, which trains an inference model jointly with a latent-conditional
policy. At test time, the agent alternates between updating its belief over the
latent and acting under the belief. We show in theory and practice that this
algorithm converges to the correct interventional policy, solves the
confounding issue, and can under certain assumptions achieve an asymptotically
optimal imitation performance.
Related papers
- Sequential Causal Imitation Learning with Unobserved Confounders [82.22545916247269]
"Monkey see monkey do" is an age-old adage, referring to na"ive imitation without a deep understanding of a system's underlying mechanics.
This paper investigates the problem of causal imitation learning in sequential settings, where the imitator must make multiple decisions per episode.
arXiv Detail & Related papers (2022-08-12T13:53:23Z) - Causal Imitation Learning with Unobserved Confounders [82.22545916247269]
We study imitation learning when sensory inputs of the learner and the expert differ.
We show that imitation could still be feasible by exploiting quantitative knowledge of the expert trajectories.
arXiv Detail & Related papers (2022-08-12T13:29:53Z) - On Covariate Shift of Latent Confounders in Imitation and Reinforcement
Learning [69.48387059607387]
We consider the problem of using expert data with unobserved confounders for imitation and reinforcement learning.
We analyze the limitations of learning from confounded expert data with and without external reward.
We validate our claims empirically on challenging assistive healthcare and recommender system simulation tasks.
arXiv Detail & Related papers (2021-10-13T07:31:31Z) - A Low Rank Promoting Prior for Unsupervised Contrastive Learning [108.91406719395417]
We construct a novel probabilistic graphical model that effectively incorporates the low rank promoting prior into the framework of contrastive learning.
Our hypothesis explicitly requires that all the samples belonging to the same instance class lie on the same subspace with small dimension.
Empirical evidences show that the proposed algorithm clearly surpasses the state-of-the-art approaches on multiple benchmarks.
arXiv Detail & Related papers (2021-08-05T15:58:25Z) - Provable Hierarchical Imitation Learning via EM [2.864550757598007]
We consider learning an options-type hierarchical policy from expert demonstrations.
We characterize the EM approach proposed by Daniel et al.
We prove that the proposed algorithm converges with high probability to a norm ball around the true parameter.
arXiv Detail & Related papers (2020-10-07T03:21:57Z) - Non-Adversarial Imitation Learning and its Connections to Adversarial
Methods [21.89749623434729]
We present a framework for non-adversarial imitation learning.
The resulting algorithms are similar to their adversarial counterparts.
We also show that our non-adversarial formulation can be used to derive novel algorithms.
arXiv Detail & Related papers (2020-08-08T13:43:06Z) - Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with
Latent Confounders [62.54431888432302]
We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders.
We show how, given only a latent variable model for states and actions, policy value can be identified from off-policy data.
arXiv Detail & Related papers (2020-07-27T22:19:01Z) - Strictly Batch Imitation Learning by Energy-based Distribution Matching [104.33286163090179]
Consider learning a policy purely on the basis of demonstrated behavior -- that is, with no access to reinforcement signals, no knowledge of transition dynamics, and no further interaction with the environment.
One solution is simply to retrofit existing algorithms for apprenticeship learning to work in the offline setting.
But such an approach leans heavily on off-policy evaluation or offline model estimation, and can be indirect and inefficient.
We argue that a good solution should be able to explicitly parameterize a policy, implicitly learn from rollout dynamics, and operate in an entirely offline fashion.
arXiv Detail & Related papers (2020-06-25T03:27:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.