Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering

• URL: http://arxiv.org/abs/2302.14039v1
• Date: Mon, 27 Feb 2023 18:51:29 GMT
• Title: Image-based Pose Estimation and Shape Reconstruction for Robot Manipulators and Soft, Continuum Robots via Differentiable Rendering
• Authors: Jingpei Lu, Fei Liu, Cedric Girerd, Michael C. Yip
• Abstract summary: State estimation from measured data is crucial for robotic applications as autonomous systems rely on sensors to capture the motion and localize in the 3D world. In this work, we achieve image-based robot pose estimation and shape reconstruction from camera images. We demonstrate that our method of using geometrical shape primitives can achieve high accuracy in shape reconstruction for a soft continuum robot and pose estimation for a robot manipulator.
• Score: 20.62295718847247
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State estimation from measured data is crucial for robotic applications as autonomous systems rely on sensors to capture the motion and localize in the 3D world. Among sensors that are designed for measuring a robot's pose, or for soft robots, their shape, vision sensors are favorable because they are information-rich, easy to set up, and cost-effective. With recent advancements in computer vision, deep learning-based methods no longer require markers for identifying feature points on the robot. However, learning-based methods are data-hungry and hence not suitable for soft and prototyping robots, as building such bench-marking datasets is usually infeasible. In this work, we achieve image-based robot pose estimation and shape reconstruction from camera images. Our method requires no precise robot meshes, but rather utilizes a differentiable renderer and primitive shapes. It hence can be applied to robots for which CAD models might not be available or are crude. Our parameter estimation pipeline is fully differentiable. The robot shape and pose are estimated iteratively by back-propagating the image loss to update the parameters. We demonstrate that our method of using geometrical shape primitives can achieve high accuracy in shape reconstruction for a soft continuum robot and pose estimation for a robot manipulator.

Related papers

• GISR: Geometric Initialization and Silhouette-based Refinement for Single-View Robot Pose and Configuration Estimation [0.0]
  GISR is a method for deep configuration and robot-to-camera pose estimation that prioritizes real-time execution. We evaluate our method on a publicly available dataset and show that GISR performs competitively with existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-05-08T08:39:25Z)
• Track2Act: Predicting Point Tracks from Internet Videos enables Diverse Zero-shot Robot Manipulation [65.46610405509338]
  Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal. We use these 2D track predictions to infer a sequence of rigid transforms of the object to be manipulated, and obtain robot end-effector poses. We show that this approach of combining scalably learned track prediction with a residual policy enables zero-shot robot manipulation.
  arXiv  Detail & Related papers  (2024-05-02T17:56:55Z)
• Real-time Holistic Robot Pose Estimation with Unknown States [30.41806081818826]
  Estimating robot pose from RGB images is a crucial problem in computer vision and robotics. Previous methods presume full knowledge of robot internal states, e.g. ground-truth robot joint angles. This work introduces an efficient framework for real-time robot pose estimation from RGB images without requiring known robot states.
  arXiv  Detail & Related papers  (2024-02-08T13:12:50Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• External Camera-based Mobile Robot Pose Estimation for Collaborative Perception with Smart Edge Sensors [22.5939915003931]
  We present an approach for estimating a mobile robot's pose w.r.t. the allocentric coordinates of a network of static cameras using multi-view RGB images. The images are processed online, locally on smart edge sensors by deep neural networks to detect the robot. With the robot's pose precisely estimated, its observations can be fused into the allocentric scene model.
  arXiv  Detail & Related papers  (2023-03-07T11:03:33Z)
• Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfer [26.21320177775571]
  We propose an end-to-end pose estimation framework that is capable of online camera-to-robot calibration and a self-supervised training method. Our framework combines deep learning and geometric vision for solving the robot pose, and the pipeline is fully differentiable.
  arXiv  Detail & Related papers  (2023-02-28T05:55:42Z)
• Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
  We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
  arXiv  Detail & Related papers  (2022-11-16T16:26:48Z)
• Neural Scene Representation for Locomotion on Structured Terrain [56.48607865960868]
  We propose a learning-based method to reconstruct the local terrain for a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot's trajectory, the estimates the topography in the robot's vicinity. We propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement.
  arXiv  Detail & Related papers  (2022-06-16T10:45:17Z)
• Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? [54.442692221567796]
  Task specification is critical for engagement of non-expert end-users and adoption of personalized robots. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use.
  arXiv  Detail & Related papers  (2022-04-23T19:39:49Z)
• Single-view robot pose and joint angle estimation via render & compare [40.05546237998603]
  We introduce RoboPose, a method to estimate the joint angles and the 6D camera-to-robot pose of a known articulated robot from a single RGB image. This is an important problem to grant mobile and itinerant autonomous systems the ability to interact with other robots.
  arXiv  Detail & Related papers  (2021-04-19T14:48:29Z)
• Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots [129.46920552019247]
  We propose the use of a Convolution Neural Network (CNN) to segment the robot hand from an image in an egocentric view. We fine-tuned the Mask-RCNN network for the specific task of segmenting the hand of the humanoid robot Vizzy.
  arXiv  Detail & Related papers  (2021-02-09T10:34:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.