Divide and Repair: Using Options to Improve Performance of Imitation Learning Against Adversarial Demonstrations

• URL: http://arxiv.org/abs/2306.04581v2
• Date: Fri, 9 Jun 2023 21:39:45 GMT
• Title: Divide and Repair: Using Options to Improve Performance of Imitation Learning Against Adversarial Demonstrations
• Authors: Prithviraj Dasgupta
• Abstract summary: We consider the problem of learning to perform a task from demonstrations given by teachers or experts. Some of the experts' demonstrations might be adversarial and demonstrate an incorrect way to perform the task. We propose a novel technique that can identify parts of demonstrated trajectories that have not been significantly modified by the adversary.
• Score: 0.6853165736531939
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We consider the problem of learning to perform a task from demonstrations given by teachers or experts, when some of the experts' demonstrations might be adversarial and demonstrate an incorrect way to perform the task. We propose a novel technique that can identify parts of demonstrated trajectories that have not been significantly modified by the adversary and utilize them for learning, using temporally extended policies or options. We first define a trajectory divergence measure based on the spatial and temporal features of demonstrated trajectories to detect and discard parts of the trajectories that have been significantly modified by an adversarial expert, and, could degrade the learner's performance, if used for learning, We then use an options-based algorithm that partitions trajectories and learns only from the parts of trajectories that have been determined as admissible. We provide theoretical results of our technique to show that repairing partial trajectories improves the sample efficiency of the demonstrations without degrading the learner's performance. We then evaluate the proposed algorithm for learning to play an Atari-like, computer-based game called LunarLander in the presence of different types and degrees of adversarial attacks of demonstrated trajectories. Our experimental results show that our technique can identify adversarially modified parts of the demonstrated trajectories and successfully prevent the learning performance from degrading due to adversarial demonstrations.

Related papers

• "Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrations [3.637365301757111]
  Methods like Reinforcement Learning from Expert Demonstrations (RLED) introduce external expert demonstrations to facilitate agent exploration during the learning process. How to select the best set of human demonstrations that is most beneficial for learning becomes a major concern. This paper presents EARLY, an algorithm that enables a learning agent to generate optimized queries of expert demonstrations in a trajectory-based feature space.
  arXiv  Detail & Related papers  (2024-06-05T08:52:21Z)
• DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning [75.68193159293425]
  In-context learning (ICL) allows transformer-based language models to learn a specific task with a few "task demonstrations" without updating their parameters. We propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.
  arXiv  Detail & Related papers  (2024-05-22T15:52:52Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Sample-efficient Adversarial Imitation Learning [45.400080101596956]
  We propose a self-supervised representation-based adversarial imitation learning method to learn state and action representations. We show a 39% relative improvement over existing adversarial imitation learning methods on MuJoCo in a setting limited to 100 expert state-action pairs.
  arXiv  Detail & Related papers  (2023-03-14T12:36:01Z)
• Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning [48.595574101874575]
  In the real world, expert demonstrations are more likely to be imperfect. A positive-unlabeled adversarial imitation learning algorithm is developed. Agent policy will be optimized to cheat the discriminator and produce trajectories similar to those optimal expert demonstrations.
  arXiv  Detail & Related papers  (2023-02-13T11:26:44Z)
• The Enemy of My Enemy is My Friend: Exploring Inverse Adversaries for Improving Adversarial Training [72.39526433794707]
  Adversarial training and its variants have been shown to be the most effective approaches to defend against adversarial examples. We propose a novel adversarial training scheme that encourages the model to produce similar outputs for an adversarial example and its inverse adversarial'' counterpart. Our training method achieves state-of-the-art robustness as well as natural accuracy.
  arXiv  Detail & Related papers  (2022-11-01T15:24:26Z)
• Exploiting Fairness to Enhance Sensitive Attributes Reconstruction [0.0]
  In recent years, a growing body of work has emerged on how to learn machine learning models under fairness constraints. We show that information about this model's fairness can be exploited by the adversary to enhance his reconstruction of the sensitive attributes of the training data. We propose a generic reconstruction correction method, which takes as input an initial guess and corrects it to comply with some user-defined constraints.
  arXiv  Detail & Related papers  (2022-09-02T06:15:15Z)
• Learn to Exceed: Stereo Inverse Reinforcement Learning with Concurrent Policy Optimization [1.0965065178451106]
  We study the problem of obtaining a control policy that can mimic and then outperform expert demonstrations in Markov decision processes. One main relevant approach is the inverse reinforcement learning (IRL), which mainly focuses on inferring a reward function from expert demonstrations. We propose a novel method that enables the learning agent to outperform the demonstrator via a new concurrent reward and action policy learning approach.
  arXiv  Detail & Related papers  (2020-09-21T02:16:21Z)
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them [57.957466608543676]
  We analyze the influence of adversarial training on the loss landscape of machine learning models. We show that the adversarial loss landscape is less favorable to optimization, due to increased curvature and more scattered gradients.
  arXiv  Detail & Related papers  (2020-06-15T13:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.