Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer

• URL: http://arxiv.org/abs/2010.08054v3
• Date: Tue, 8 Feb 2022 00:26:18 GMT
• Title: Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer
• Authors: Jingpei Lu, Florian Richter, Michael Yip
• Abstract summary: Keypoint detection is an essential building block for many robotic applications. Deep learning methods have the ability to detect user-defined keypoints in a marker-less manner. We propose a new and autonomous way to define the keypoint locations that overcomes these challenges.
• Score: 10.369766652751169
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Keypoint detection is an essential building block for many robotic applications like motion capture and pose estimation. Historically, keypoints are detected using uniquely engineered markers such as checkerboards or fiducials. More recently, deep learning methods have been explored as they have the ability to detect user-defined keypoints in a marker-less manner. However, different manually selected keypoints can have uneven performance when it comes to detection and localization. An example of this can be found on symmetric robotic tools where DNN detectors cannot solve the correspondence problem correctly. In this work, we propose a new and autonomous way to define the keypoint locations that overcomes these challenges. The approach involves finding the optimal set of keypoints on robotic manipulators for robust visual detection and localization. Using a robotic simulator as a medium, our algorithm utilizes synthetic data for DNN training, and the proposed algorithm is used to optimize the selection of keypoints through an iterative approach. The results show that when using the optimized keypoints, the detection performance of the DNNs improved significantly. We further use the optimized keypoints for real robotic applications by using domain randomization to bridge the reality gap between the simulator and the physical world. The physical world experiments show how the proposed method can be applied to the wide-breadth of robotic applications that require visual feedback, such as camera-to-robot calibration, robotic tool tracking, and end-effector pose estimation.

Related papers

• Keypoint Abstraction using Large Models for Object-Relative Imitation Learning [78.92043196054071]
  Generalization to novel object configurations and instances across diverse tasks and environments is a critical challenge in robotics. Keypoint-based representations have been proven effective as a succinct representation for essential object capturing features. We propose KALM, a framework that leverages large pre-trained vision-language models to automatically generate task-relevant and cross-instance consistent keypoints.
  arXiv  Detail & Related papers  (2024-10-30T17:37:31Z)
• Kalib: Markerless Hand-Eye Calibration with Keypoint Tracking [52.4190876409222]
  Hand-eye calibration involves estimating the transformation between the camera and the robot. Recent advancements in deep learning offer markerless techniques, but they present challenges. We propose Kalib, an automatic and universal markerless hand-eye calibration pipeline.
  arXiv  Detail & Related papers  (2024-08-20T06:03:40Z)
• Learning to Make Keypoints Sub-Pixel Accurate [80.55676599677824]
  This work addresses the challenge of sub-pixel accuracy in detecting 2D local features. We propose a novel network that enhances any detector with sub-pixel precision by learning an offset vector for detected features.
  arXiv  Detail & Related papers  (2024-07-16T12:39:56Z)
• HPointLoc: Point-based Indoor Place Recognition using Synthetic RGB-D Images [58.720142291102135]
  We present a novel dataset named as HPointLoc, specially designed for exploring capabilities of visual place recognition in indoor environment. The dataset is based on the popular Habitat simulator, in which it is possible to generate indoor scenes using both own sensor data and open datasets.
  arXiv  Detail & Related papers  (2022-12-30T12:20:56Z)
• Centroid Distance Keypoint Detector for Colored Point Clouds [32.74803728070627]
  Keypoint detection serves as the basis for many computer vision and robotics applications. We propose an efficient multi-modal keypoint detector that can extract both geometry-salient and color-salient keypoints in colored point clouds.
  arXiv  Detail & Related papers  (2022-10-04T00:55:51Z)
• Robot Self-Calibration Using Actuated 3D Sensors [0.0]
  This paper treats robot calibration as an offline SLAM problem, where scanning poses are linked to a fixed point in space by a moving kinematic chain. As such, the presented framework allows robot calibration using nothing but an arbitrary eye-in-hand depth sensor. A detailed evaluation of the system is shown on a real robot with various attached 3D sensors.
  arXiv  Detail & Related papers  (2022-06-07T16:35:08Z)
• Bayesian Imitation Learning for End-to-End Mobile Manipulation [80.47771322489422]
  Augmenting policies with additional sensor inputs, such as RGB + depth cameras, is a straightforward approach to improving robot perception capabilities. We show that using the Variational Information Bottleneck to regularize convolutional neural networks improves generalization to held-out domains. We demonstrate that our method is able to help close the sim-to-real gap and successfully fuse RGB and depth modalities.
  arXiv  Detail & Related papers  (2022-02-15T17:38:30Z)
• End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation [7.374994747693731]
  We present an end-to-end Reinforcement Learning framework for robotic manipulation tasks, using a robust and efficient keypoints representation. The proposed method learns keypoints from camera images as the state representation, through a self-supervised autoencoder architecture. We demonstrate the effectiveness of the proposed method on robotic manipulation tasks including grasping and pushing, in different scenarios.
  arXiv  Detail & Related papers  (2022-02-12T09:58:09Z)
• CNN-based Omnidirectional Object Detection for HermesBot Autonomous Delivery Robot with Preliminary Frame Classification [53.56290185900837]
  We propose an algorithm for optimizing a neural network for object detection using preliminary binary frame classification. An autonomous mobile robot with 6 rolling-shutter cameras on the perimeter providing a 360-degree field of view was used as the experimental setup.
  arXiv  Detail & Related papers  (2021-10-22T15:05:37Z)
• Real-Time Object Detection and Recognition on Low-Compute Humanoid Robots using Deep Learning [0.12599533416395764]
  We describe a novel architecture that enables multiple low-compute NAO robots to perform real-time detection, recognition and localization of objects in its camera view. The proposed algorithm for object detection and localization is an empirical modification of YOLOv3, based on indoor experiments in multiple scenarios. The architecture also comprises of an effective end-to-end pipeline to feed the real-time frames from the camera feed to the neural net and use its results for guiding the robot.
  arXiv  Detail & Related papers  (2020-01-20T05:24:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.