Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects
- URL: http://arxiv.org/abs/2110.14217v1
- Date: Wed, 27 Oct 2021 07:02:53 GMT
- Title: Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects
- Authors: Jeffrey Ichnowski, Yahav Avigal, Justin Kerr and Ken Goldberg
- Abstract summary: Existing depth cameras have difficulty detecting, localizing, and inferring the geometry of transparent objects.
We propose using neural radiance fields (NeRF) to detect, localize, and infer the geometry of transparent objects.
We show that NeRF and Dex-Net are able to reliably compute robust grasps on transparent objects, achieving 90% and 100% grasp success rates in physical experiments on an ABB YuMi.
- Score: 23.933258829652186
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The ability to grasp and manipulate transparent objects is a major challenge
for robots. Existing depth cameras have difficulty detecting, localizing, and
inferring the geometry of such objects. We propose using neural radiance fields
(NeRF) to detect, localize, and infer the geometry of transparent objects with
sufficient accuracy to find and grasp them securely. We leverage NeRF's
view-independent learned density, place lights to increase specular
reflections, and perform a transparency-aware depth-rendering that we feed into
the Dex-Net grasp planner. We show how additional lights create specular
reflections that improve the quality of the depth map, and test a setup for a
robot workcell equipped with an array of cameras to perform transparent object
manipulation. We also create synthetic and real datasets of transparent objects
in real-world settings, including singulated objects, cluttered tables, and the
top rack of a dishwasher. In each setting we show that NeRF and Dex-Net are
able to reliably compute robust grasps on transparent objects, achieving 90%
and 100% grasp success rates in physical experiments on an ABB YuMi, on objects
where baseline methods fail.
Related papers
- ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation [18.140839442955485]
We develop a vision transformer-based algorithm for stereo depth recovery of transparent objects.
Our method incorporates a parameter-aligned, domain-adaptive, and physically realistic Sim2Real simulation for efficient data generation.
Our experimental results demonstrate the model's exceptional Sim2Real generalizability in real-world scenarios.
arXiv Detail & Related papers (2024-09-13T15:44:38Z) - Residual-NeRF: Learning Residual NeRFs for Transparent Object Manipulation [7.395916591967461]
Existing methods have difficulty reconstructing complete depth maps for challenging transparent objects.
Recent work has shown neural radiance fields (NeRFs) work well for depth perception in scenes with transparent objects.
We propose Residual-NeRF, a method to improve depth perception and training speed for transparent objects.
arXiv Detail & Related papers (2024-05-10T01:53:29Z) - ASGrasp: Generalizable Transparent Object Reconstruction and Grasping from RGB-D Active Stereo Camera [9.212504138203222]
We propose ASGrasp, a 6-DoF grasp detection network that uses an RGB-D active stereo camera.
Our system distinguishes itself by its ability to directly utilize raw IR and RGB images for transparent object geometry reconstruction.
Our experiments demonstrate that ASGrasp can achieve over 90% success rate for generalizable transparent object grasping.
arXiv Detail & Related papers (2024-05-09T09:44:51Z) - Closing the Visual Sim-to-Real Gap with Object-Composable NeRFs [59.12526668734703]
We introduce Composable Object Volume NeRF (COV-NeRF), an object-composable NeRF model that is the centerpiece of a real-to-sim pipeline.
COV-NeRF extracts objects from real images and composes them into new scenes, generating photorealistic renderings and many types of 2D and 3D supervision.
arXiv Detail & Related papers (2024-03-07T00:00:02Z) - RFTrans: Leveraging Refractive Flow of Transparent Objects for Surface
Normal Estimation and Manipulation [50.10282876199739]
This paper introduces RFTrans, an RGB-D-based method for surface normal estimation and manipulation of transparent objects.
It integrates the RFNet, which predicts refractive flow, object mask, and boundaries, followed by the F2Net, which estimates surface normal from the refractive flow.
A real-world robot grasping task witnesses an 83% success rate, proving that refractive flow can help enable direct sim-to-real transfer.
arXiv Detail & Related papers (2023-11-21T07:19:47Z) - MonoGraspNet: 6-DoF Grasping with a Single RGB Image [73.96707595661867]
6-DoF robotic grasping is a long-lasting but unsolved problem.
Recent methods utilize strong 3D networks to extract geometric grasping representations from depth sensors.
We propose the first RGB-only 6-DoF grasping pipeline called MonoGraspNet.
arXiv Detail & Related papers (2022-09-26T21:29:50Z) - NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance
Fields [54.27264716713327]
We show that a Neural Radiance Fields (NeRF) representation of a scene can be used to train dense object descriptors.
We use an optimized NeRF to extract dense correspondences between multiple views of an object, and then use these correspondences as training data for learning a view-invariant representation of the object.
Dense correspondence models supervised with our method significantly outperform off-the-shelf learned descriptors by 106%.
arXiv Detail & Related papers (2022-03-03T18:49:57Z) - TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth
Completion and Grasping [46.6058840385155]
We contribute a large-scale real-world dataset for transparent object depth completion.
Our dataset contains 57,715 RGB-D images from 130 different scenes.
We propose an end-to-end depth completion network, which takes the RGB image and the inaccurate depth map as inputs and outputs a refined depth map.
arXiv Detail & Related papers (2022-02-17T06:50:20Z) - Seeing Glass: Joint Point Cloud and Depth Completion for Transparent
Objects [16.714074893209713]
TranspareNet is a joint point cloud and depth completion method.
It can complete the depth of transparent objects in cluttered and complex scenes.
TranspareNet outperforms existing state-of-the-art depth completion methods on multiple datasets.
arXiv Detail & Related papers (2021-09-30T21:09:09Z) - Through the Looking Glass: Neural 3D Reconstruction of Transparent
Shapes [75.63464905190061]
Complex light paths induced by refraction and reflection have prevented both traditional and deep multiview stereo from solving this problem.
We propose a physically-based network to recover 3D shape of transparent objects using a few images acquired with a mobile phone camera.
Our experiments show successful recovery of high-quality 3D geometry for complex transparent shapes using as few as 5-12 natural images.
arXiv Detail & Related papers (2020-04-22T23:51:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.