Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddings

• URL: http://arxiv.org/abs/2112.04910v2
• Date: Mon, 13 Dec 2021 11:39:01 GMT
• Title: Few-Shot Keypoint Detection as Task Adaptation via Latent Embeddings
• Authors: Mel Vecerik and Jackie Kay and Raia Hadsell and Lourdes Agapito and Jon Scholz
• Abstract summary: Existing approaches either compute dense keypoint embeddings in a single forward pass, or allocate their full capacity to a sparse set of points. In this paper we explore a middle ground based on the observation that the number of relevant points at a given time are typically relatively few. Our main contribution is a novel architecture, inspired by few-shot task adaptation, which allows a sparse-style network to condition on a keypoint embedding.
• Score: 17.04471874483516
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dense object tracking, the ability to localize specific object points with pixel-level accuracy, is an important computer vision task with numerous downstream applications in robotics. Existing approaches either compute dense keypoint embeddings in a single forward pass, meaning the model is trained to track everything at once, or allocate their full capacity to a sparse predefined set of points, trading generality for accuracy. In this paper we explore a middle ground based on the observation that the number of relevant points at a given time are typically relatively few, e.g. grasp points on a target object. Our main contribution is a novel architecture, inspired by few-shot task adaptation, which allows a sparse-style network to condition on a keypoint embedding that indicates which point to track. Our central finding is that this approach provides the generality of dense-embedding models, while offering accuracy significantly closer to sparse-keypoint approaches. We present results illustrating this capacity vs. accuracy trade-off, and demonstrate the ability to zero-shot transfer to new object instances (within-class) using a real-robot pick-and-place task.

Related papers

• Keypoint Abstraction using Large Models for Object-Relative Imitation Learning [78.92043196054071]
  Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for essential object capturing features. We propose KALM, a framework that leverages large pre-trained vision-language models to automatically generate task-relevant and cross-instance consistent keypoints.
  arXiv  Detail & Related papers  (2024-10-30T17:37:31Z)
• Local Occupancy-Enhanced Object Grasping with Multiple Triplanar Projection [24.00828999360765]
  This paper addresses the challenge of robotic grasping of general objects. The proposed model first runs by proposing a number of most likely grasp points in the scene. Around each grasp point, a module is designed to infer any voxel in its neighborhood to be either void or occupied by some object. The model further estimates 6-DoF grasp poses utilizing the local occupancy-enhanced object shape information.
  arXiv  Detail & Related papers  (2024-07-22T16:22:28Z)
• DeTra: A Unified Model for Object Detection and Trajectory Forecasting [68.85128937305697]
  Our approach formulates the union of the two tasks as a trajectory refinement problem. To tackle this unified task, we design a refinement transformer that infers the presence, pose, and multi-modal future behaviors of objects. In our experiments, we observe that ourmodel outperforms the state-of-the-art on Argoverse 2 Sensor and Open dataset.
  arXiv  Detail & Related papers  (2024-06-06T18:12:04Z)
• Weakly-Supervised Cross-Domain Segmentation of Electron Microscopy with Sparse Point Annotation [1.124958340749622]
  We introduce a multitask learning framework to leverage correlations among the counting, detection, and segmentation tasks. We develop a cross-position cut-and-paste for label augmentation and an entropy-based pseudo-label selection. The proposed model is capable of significantly outperforming UDA methods and produces comparable performance as the supervised counterpart.
  arXiv  Detail & Related papers  (2024-03-31T12:22:23Z)
• SASA: Semantics-Augmented Set Abstraction for Point-based 3D Object Detection [78.90102636266276]
  We propose a novel set abstraction method named Semantics-Augmented Set Abstraction (SASA) Based on the estimated point-wise foreground scores, we then propose a semantics-guided point sampling algorithm to help retain more important foreground points during down-sampling. In practice, SASA shows to be effective in identifying valuable points related to foreground objects and improving feature learning for point-based 3D detection.
  arXiv  Detail & Related papers  (2022-01-06T08:54:47Z)
• Rethinking Keypoint Representations: Modeling Keypoints and Poses as Objects for Multi-Person Human Pose Estimation [79.78017059539526]
  We propose a new heatmap-free keypoint estimation method in which individual keypoints and sets of spatially related keypoints (i.e., poses) are modeled as objects within a dense single-stage anchor-based detection framework. In experiments, we observe that KAPAO is significantly faster and more accurate than previous methods, which suffer greatly from heatmap post-processing. Our large model, KAPAO-L, achieves an AP of 70.6 on the Microsoft COCO Keypoints validation set without test-time augmentation.
  arXiv  Detail & Related papers  (2021-11-16T15:36:44Z)
• S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency [11.357804868755155]
  We advocate semantic 3D keypoints as a visual representation, and present a semi-supervised training objective. Unlike local texture-based approaches, our model integrates contextual information from a large area. We demonstrate that this ability to locate semantic keypoints enables high level scripting of human understandable behaviours.
  arXiv  Detail & Related papers  (2020-09-30T14:44:54Z)
• A Self-Training Approach for Point-Supervised Object Detection and Counting in Crowds [54.73161039445703]
  We propose a novel self-training approach that enables a typical object detector trained only with point-level annotations. During training, we utilize the available point annotations to supervise the estimation of the center points of objects. Experimental results show that our approach significantly outperforms state-of-the-art point-supervised methods under both detection and counting tasks.
  arXiv  Detail & Related papers  (2020-07-25T02:14:42Z)
• Point-Set Anchors for Object Detection, Instance Segmentation and Pose Estimation [85.96410825961966]
  We argue that the image features extracted at a central point contain limited information for predicting distant keypoints or bounding box boundaries. To facilitate inference, we propose to instead perform regression from a set of points placed at more advantageous positions. We apply this proposed framework, called Point-Set Anchors, to object detection, instance segmentation, and human pose estimation.
  arXiv  Detail & Related papers  (2020-07-06T15:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.