Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning

• URL: http://arxiv.org/abs/2410.15910v2
• Date: Tue, 22 Oct 2024 05:06:36 GMT
• Title: Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning
• Authors: Hanlin Yang, Jian Yao, Weiming Liu, Qing Wang, Hanmin Qin, Hansheng Kong, Kirk Tang, Jiechao Xiong, Chao Yu, Kai Li, Junliang Xing, Hongwu Chen, Juchao Zhuo, Qiang Fu, Yang Wei, Haobo Fu,
• Abstract summary: This paper presents a new principled method in recovering diverse polices from expert data. In particular, after inferring or assigning a latent style for a trajectory, we enhance the vanilla behavioral cloning by incorporating a weighting mechanism. We provide theoretical justifications for our new objective, and extensive empirical evaluations confirm the effectiveness of our method.
• Score: 24.600297554387108
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recovering a spectrum of diverse policies from a set of expert trajectories is an important research topic in imitation learning. After determining a latent style for a trajectory, previous diverse policies recovering methods usually employ a vanilla behavioral cloning learning objective conditioned on the latent style, treating each state-action pair in the trajectory with equal importance. Based on an observation that in many scenarios, behavioral styles are often highly relevant with only a subset of state-action pairs, this paper presents a new principled method in diverse polices recovery. In particular, after inferring or assigning a latent style for a trajectory, we enhance the vanilla behavioral cloning by incorporating a weighting mechanism based on pointwise mutual information. This additional weighting reflects the significance of each state-action pair's contribution to learning the style, thus allowing our method to focus on state-action pairs most representative of that style. We provide theoretical justifications for our new objective, and extensive empirical evaluations confirm the effectiveness of our method in recovering diverse policies from expert data.

Related papers

• Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy For Visuomotor Imitation Learning [2.6696199945489534]
  We propose an object-centric recovery policy framework to address the challenges of out-of-distribution scenarios in visuomotor policy learning. We demonstrate the effectiveness of our framework in both simulation and real robot experiments.
  arXiv  Detail & Related papers  (2024-11-05T17:41:14Z)
• Inverse Reinforcement Learning from Non-Stationary Learning Agents [11.203097744443898]
  We study an inverse reinforcement learning problem that involves learning the reward function of a learning agent using trajectory data collected while this agent is learning its optimal policy. We propose an inverse reinforcement learning method that allows us to estimate the policy parameters of the learning agent which can then be used to estimate its reward function.
  arXiv  Detail & Related papers  (2024-10-18T03:02:44Z)
• Invariant Causal Imitation Learning for Generalizable Policies [87.51882102248395]
  We propose Invariant Causal Learning (ICIL) to learn an imitation policy. ICIL learns a representation of causal features that is disentangled from the specific representations of noise variables. We show that ICIL is effective in learning imitation policies capable of generalizing to unseen environments.
  arXiv  Detail & Related papers  (2023-11-02T16:52:36Z)
• Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning [53.97273491846883]
  We propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-08-28T20:46:07Z)
• Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling [34.88897402357158]
  We show that due to the limited distributional expressivity of policy models, previous methods might still select unseen actions during training. We adopt a generative approach by decoupling the learned policy into two parts: an expressive generative behavior model and an action evaluation model. Our proposed method achieves competitive or superior performance compared with state-of-the-art offline RL methods.
  arXiv  Detail & Related papers  (2022-09-29T04:36:23Z)
• Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates [63.58053355357644]
  We study the problem of learning a good set of policies, so that when combined together, they can solve a wide variety of unseen reinforcement learning tasks. We show theoretically that having access to a specific set of diverse policies, which we call a set of independent policies, can allow for instantaneously achieving high-level performance.
  arXiv  Detail & Related papers  (2021-12-30T12:20:46Z)
• Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning [131.1852444489217]
  This paper presents Object-aware REgularizatiOn (OREO), a technique that regularizes an imitation policy in an object-aware manner. Our main idea is to encourage a policy to uniformly attend to all semantic objects, in order to prevent the policy from exploiting nuisance variables strongly correlated with expert actions.
  arXiv  Detail & Related papers  (2021-10-27T01:56:23Z)
• Fully Unsupervised Person Re-identification viaSelective Contrastive Learning [58.5284246878277]
  Person re-identification (ReID) aims at searching the same identity person among images captured by various cameras. We propose a novel selective contrastive learning framework for unsupervised feature learning. Experimental results demonstrate the superiority of our method in unsupervised person ReID compared with the state-of-the-arts.
  arXiv  Detail & Related papers  (2020-10-15T09:09:23Z)
• Reward-Conditioned Policies [100.64167842905069]
  imitation learning requires near-optimal expert data. Can we learn effective policies via supervised learning without demonstrations? We show how such an approach can be derived as a principled method for policy search.
  arXiv  Detail & Related papers  (2019-12-31T18:07:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.