DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation

• URL: http://arxiv.org/abs/2107.08067v1
• Date: Fri, 16 Jul 2021 18:20:58 GMT
• Title: DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation
• Authors: Bao Thach, Alan Kuntz, Tucker Hermans
• Abstract summary: We propose a novel approach to 3D deformable object manipulation leveraging a deep neural network called DeformerNet. We explicitly use 3D point clouds as the state representation and apply Convolutional Neural Network on point clouds to learn the 3D features. Once trained in an end-to-end fashion, DeformerNet directly maps the current point cloud of a deformable object, as well as a target point cloud shape, to the desired displacement in robot gripper position.
• Score: 5.733365759103406
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In this paper, we propose a novel approach to 3D deformable object manipulation leveraging a deep neural network called DeformerNet. Controlling the shape of a 3D object requires an effective state representation that can capture the full 3D geometry of the object. Current methods work around this problem by defining a set of feature points on the object or only deforming the object in 2D image space, which does not truly address the 3D shape control problem. Instead, we explicitly use 3D point clouds as the state representation and apply Convolutional Neural Network on point clouds to learn the 3D features. These features are then mapped to the robot end-effector's position using a fully-connected neural network. Once trained in an end-to-end fashion, DeformerNet directly maps the current point cloud of a deformable object, as well as a target point cloud shape, to the desired displacement in robot gripper position. In addition, we investigate the problem of predicting the manipulation point location given the initial and goal shape of the object.

Related papers

• DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects [13.138509669247508]
  Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the manipulated object and a point cloud of the goal shape. This shape embedding enables the robot to learn a visual servo controller that computes the desired robot end-effector action to
  arXiv  Detail & Related papers  (2023-05-08T04:08:06Z)
• Neural Correspondence Field for Object Pose Estimation [67.96767010122633]
  We propose a method for estimating the 6DoF pose of a rigid object with an available 3D model from a single RGB image. Unlike classical correspondence-based methods which predict 3D object coordinates at pixels of the input image, the proposed method predicts 3D object coordinates at 3D query points sampled in the camera frustum.
  arXiv  Detail & Related papers  (2022-07-30T01:48:23Z)
• Points2NeRF: Generating Neural Radiance Fields from 3D point cloud [0.0]
  We propose representing 3D objects as Neural Radiance Fields (NeRFs) We leverage a hypernetwork paradigm and train the model to take a 3D point cloud with the associated color values. Our method provides efficient 3D object representation and offers several advantages over the existing approaches.
  arXiv  Detail & Related papers  (2022-06-02T20:23:33Z)
• Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds [7.1659268120093635]
  Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the object being manipulated and a point cloud of the goal shape to learn a low-dimensional representation of the object shape.
  arXiv  Detail & Related papers  (2021-10-10T02:34:57Z)
• KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control [64.46042014759671]
  KeypointDeformer is an unsupervised method for shape control through automatically discovered 3D keypoints. Our approach produces intuitive and semantically consistent control of shape deformations.
  arXiv  Detail & Related papers  (2021-04-22T17:59:08Z)
• ParaNet: Deep Regular Representation for 3D Point Clouds [62.81379889095186]
  ParaNet is a novel end-to-end deep learning framework for representing 3D point clouds. It converts an irregular 3D point cloud into a regular 2D color image, named point geometry image (PGI) In contrast to conventional regular representation modalities based on multi-view projection and voxelization, the proposed representation is differentiable and reversible.
  arXiv  Detail & Related papers  (2020-12-05T13:19:55Z)
• Interactive Annotation of 3D Object Geometry using 2D Scribbles [84.51514043814066]
  In this paper, we propose an interactive framework for annotating 3D object geometry from point cloud data and RGB imagery. Our framework targets naive users without artistic or graphics expertise.
  arXiv  Detail & Related papers  (2020-08-24T21:51:29Z)
• Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation [87.54570024320354]
  State-of-the-art methods for large-scale driving-scene LiDAR semantic segmentation often project and process the point clouds in the 2D space. A straightforward solution to tackle the issue of 3D-to-2D projection is to keep the 3D representation and process the points in the 3D space. We develop a 3D cylinder partition and a 3D cylinder convolution based framework, termed as Cylinder3D, which exploits the 3D topology relations and structures of driving-scene point clouds.
  arXiv  Detail & Related papers  (2020-08-04T13:56:19Z)
• DOPS: Learning to Detect 3D Objects and Predict their 3D Shapes [54.239416488865565]
  We propose a fast single-stage 3D object detection method for LIDAR data. The core novelty of our method is a fast, single-pass architecture that both detects objects in 3D and estimates their shapes. We find that our proposed method achieves state-of-the-art results by 5% on object detection in ScanNet scenes, and it gets top results by 3.4% in the Open dataset.
  arXiv  Detail & Related papers  (2020-04-02T17:48:50Z)
• Hypernetwork approach to generating point clouds [18.67883065951206]
  We build a hyper network that returns weights of a particular neural network trained to map points into a 3D shape. A particular 3D shape can be generated using point-by-point sampling from the assumed prior distribution. Since the hyper network is based on an auto-encoder architecture trained to reconstruct realistic 3D shapes, the target network weights can be considered a parametrization of the surface of a 3D shape.
  arXiv  Detail & Related papers  (2020-02-10T11:09:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.