Contrastively Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Grasping

• URL: http://arxiv.org/abs/2104.00878v1
• Date: Fri, 2 Apr 2021 04:18:53 GMT
• Title: Contrastively Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Grasping
• Abstract summary: We integrate affordance discovery and affordance-based policy learning in an end-to-end imitation learning framework based on deep neural networks. We propose a contrastive learning framework that consists of a Siamese encoder and a trajectory decoder. Our experimental results demonstrate that our model with the coupled triplet loss achieves the highest grasping success rate in a simulated robot environment.
• Score: 1.5727276506140881
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Conventional works that learn grasping affordance from demonstrations need to explicitly predict grasping configurations, such as gripper approaching angles or grasping preshapes. Classic motion planners could then sample trajectories by using such predicted configurations. In this work, our goal is instead to fill the gap between affordance discovery and affordance-based policy learning by integrating the two objectives in an end-to-end imitation learning framework based on deep neural networks. From a psychological perspective, there is a close association between attention and affordance. Therefore, with an end-to-end neural network, we propose to learn affordance cues as visual attention that serves as a useful indicating signal of how a demonstrator accomplishes tasks, instead of explicitly modeling affordances. To achieve this, we propose a contrastive learning framework that consists of a Siamese encoder and a trajectory decoder. We further introduce a coupled triplet loss to encourage the discovered affordance cues to be more affordance-relevant. Our experimental results demonstrate that our model with the coupled triplet loss achieves the highest grasping success rate in a simulated robot environment.

Related papers

• Learning a Neural Association Network for Self-supervised Multi-Object Tracking [34.07776597698471]
  This paper introduces a novel framework to learn data association for multi-object tracking in a self-supervised manner. Motivated by the fact that in real-world scenarios object motion can be usually represented by a Markov process, we present a novel expectation (EM) algorithm that trains a neural network to associate detections for tracking. We evaluate our approach on the challenging MOT17 and MOT20 datasets and achieve state-of-the-art results in comparison to self-supervised trackers.
  arXiv  Detail & Related papers  (2024-11-18T12:22:29Z)
• Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
  In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
  arXiv  Detail & Related papers  (2024-10-31T22:54:34Z)
• Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence [56.092059713922744]
  We show that using high-level contextualized features as prediction targets can achieve superior performance. Specifically, we propose Skeleton2vec, a simple and efficient self-supervised 3D action representation learning framework. Our proposed Skeleton2vec outperforms previous methods and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-01-01T12:08:35Z)
• Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos [71.20376514273367]
  We propose a unified point cloud video self-supervised learning framework for object-centric and scene-centric data. Our method outperforms supervised counterparts on a wide range of downstream tasks.
  arXiv  Detail & Related papers  (2023-08-18T02:17:47Z)
• Pretraining on Interactions for Learning Grounded Affordance Representations [22.290431852705662]
  We train a neural network to predict objects' trajectories in a simulated interaction. We show that our network's latent representations differentiate between both observed and unobserved affordances. Our results suggest a way in which modern deep learning approaches to grounded language learning can be integrated with traditional formal semantic notions of lexical representations.
  arXiv  Detail & Related papers  (2022-07-05T19:19:53Z)
• Correlation-Aware Deep Tracking [83.51092789908677]
  We propose a novel target-dependent feature network inspired by the self-/cross-attention scheme. Our network deeply embeds cross-image feature correlation in multiple layers of the feature network. Our model can be flexibly pre-trained on abundant unpaired images, leading to notably faster convergence than the existing methods.
  arXiv  Detail & Related papers  (2022-03-03T11:53:54Z)
• Towards Improving Embedding Based Models of Social Network Alignment via Pseudo Anchors [43.48748964787508]
  We propose a novel learning framework which tries to enforce the resulting embeddings to be more widely apart among the users. We propose a meta-learning algorithm to guide the updating of the pseudo anchor embeddings during the learning process. We have incorporated the proposed learning framework into several state-of-the-art models.
  arXiv  Detail & Related papers  (2021-11-22T16:40:32Z)
• Analytically Tractable Hidden-States Inference in Bayesian Neural Networks [0.0]
  We show how we can leverage the tractable approximate Gaussian inference's (TAGI) capabilities to infer hidden states. One novel aspect it allows is to infer hidden states through the imposition of constraints designed to achieve specific objectives.
  arXiv  Detail & Related papers  (2021-07-08T11:11:25Z)
• Variational Structured Attention Networks for Deep Visual Representation Learning [49.80498066480928]
  We propose a unified deep framework to jointly learn both spatial attention maps and channel attention in a principled manner. Specifically, we integrate the estimation and the interaction of the attentions within a probabilistic representation learning framework. We implement the inference rules within the neural network, thus allowing for end-to-end learning of the probabilistic and the CNN front-end parameters.
  arXiv  Detail & Related papers  (2021-03-05T07:37:24Z)
• Dynamic Inference: A New Approach Toward Efficient Video Action Recognition [69.9658249941149]
  Action recognition in videos has achieved great success recently, but it remains a challenging task due to the massive computational cost. We propose a general dynamic inference idea to improve inference efficiency by leveraging the variation in the distinguishability of different videos.
  arXiv  Detail & Related papers  (2020-02-09T11:09:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.