Imitation Learning with Human Eye Gaze via Multi-Objective Prediction

• URL: http://arxiv.org/abs/2102.13008v3
• Date: Sat, 22 Jul 2023 19:46:36 GMT
• Title: Imitation Learning with Human Eye Gaze via Multi-Objective Prediction
• Authors: Ravi Kumar Thakur, MD-Nazmus Samin Sunbeam, Vinicius G. Goecks, Ellen Novoseller, Ritwik Bera, Vernon J. Lawhern, Gregory M. Gremillion, John Valasek, Nicholas R. Waytowich
• Abstract summary: We propose Gaze Regularized Imitation Learning (GRIL), a novel context-aware imitation learning architecture. GRIL learns concurrently from both human demonstrations and eye gaze to solve tasks where visual attention provides important context. We show that GRIL outperforms several state-of-the-art gaze-based imitation learning algorithms, simultaneously learns to predict human visual attention, and generalizes to scenarios not present in the training data.
• Score: 3.5779268406205618
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Approaches for teaching learning agents via human demonstrations have been widely studied and successfully applied to multiple domains. However, the majority of imitation learning work utilizes only behavioral information from the demonstrator, i.e. which actions were taken, and ignores other useful information. In particular, eye gaze information can give valuable insight towards where the demonstrator is allocating visual attention, and holds the potential to improve agent performance and generalization. In this work, we propose Gaze Regularized Imitation Learning (GRIL), a novel context-aware, imitation learning architecture that learns concurrently from both human demonstrations and eye gaze to solve tasks where visual attention provides important context. We apply GRIL to a visual navigation task, in which an unmanned quadrotor is trained to search for and navigate to a target vehicle in a photorealistic simulated environment. We show that GRIL outperforms several state-of-the-art gaze-based imitation learning algorithms, simultaneously learns to predict human visual attention, and generalizes to scenarios not present in the training data. Supplemental videos and code can be found at https://sites.google.com/view/gaze-regularized-il/.

Related papers

• Voila-A: Aligning Vision-Language Models with User's Gaze Attention [56.755993500556734]
  We introduce gaze information as a proxy for human attention to guide Vision-Language Models (VLMs) We propose a novel approach, Voila-A, for gaze alignment to enhance the interpretability and effectiveness of these models in real-world applications.
  arXiv  Detail & Related papers  (2023-12-22T17:34:01Z)
• What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? [57.92924256181857]
  We find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models.
  arXiv  Detail & Related papers  (2023-11-03T18:09:08Z)
• Human-oriented Representation Learning for Robotic Manipulation [64.59499047836637]
  Humans inherently possess generalizable visual representations that empower them to efficiently explore and interact with the environments in manipulation tasks. We formalize this idea through the lens of human-oriented multi-task fine-tuning on top of pre-trained visual encoders. Our Task Fusion Decoder consistently improves the representation of three state-of-the-art visual encoders for downstream manipulation policy-learning.
  arXiv  Detail & Related papers  (2023-10-04T17:59:38Z)
• Brief Introduction to Contrastive Learning Pretext Tasks for Visual Representation [0.0]
  We introduce contrastive learning, a subset of unsupervised learning methods. The purpose of contrastive learning is to embed augmented samples from the same sample near to each other while pushing away those that are not. We offer some strategies from contrastive learning that have recently been published and are focused on pretext tasks for visual representation.
  arXiv  Detail & Related papers  (2022-10-06T18:54:10Z)
• Embodied Learning for Lifelong Visual Perception [33.02424587900808]
  We study lifelong visual perception in an embodied setup, where we develop new models and compare various agents that navigate in buildings. The purpose of the agents is to recognize objects and other semantic classes in the whole building at the end of a process that combines exploration and active visual learning.
  arXiv  Detail & Related papers  (2021-12-28T10:47:13Z)
• Playful Interactions for Representation Learning [82.59215739257104]
  We propose to use playful interactions in a self-supervised manner to learn visual representations for downstream tasks. We collect 2 hours of playful data in 19 diverse environments and use self-predictive learning to extract visual representations. Our representations generalize better than standard behavior cloning and can achieve similar performance with only half the number of required demonstrations.
  arXiv  Detail & Related papers  (2021-07-19T17:54:48Z)
• Curious Representation Learning for Embodied Intelligence [81.21764276106924]
  Self-supervised representation learning has achieved remarkable success in recent years. Yet to build truly intelligent agents, we must construct representation learning algorithms that can learn from environments. We propose a framework, curious representation learning, which jointly learns a reinforcement learning policy and a visual representation model.
  arXiv  Detail & Related papers  (2021-05-03T17:59:20Z)
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions [50.435861435121915]
  We use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. Our experiments show that our "muscly-supervised" representation outperforms a visual-only state-of-the-art method MoCo.
  arXiv  Detail & Related papers  (2020-10-16T17:46:53Z)
• Active Perception and Representation for Robotic Manipulation [0.8315801422499861]
  We present a framework that leverages the benefits of active perception to accomplish manipulation tasks. Our agent uses viewpoint changes to localize objects, to learn state representations in a self-supervised manner, and to perform goal-directed actions. Compared to vanilla deep Q-learning algorithms, our model is at least four times more sample-efficient.
  arXiv  Detail & Related papers  (2020-03-15T01:43:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.