Fast Point Cloud to Mesh Reconstruction for Deformable Object Tracking

• URL: http://arxiv.org/abs/2311.02749v3
• Date: Tue, 26 Mar 2024 21:42:34 GMT
• Title: Fast Point Cloud to Mesh Reconstruction for Deformable Object Tracking
• Authors: Elham Amin Mansour, Hehui Zheng, Robert K. Katzschmann,
• Abstract summary: We develop a method that takes as input a template mesh which is the mesh of an object in its non-deformed state and a deformed point cloud of the same object. Our trained model can perform mesh reconstruction and tracking at a rate of 58Hz on a template mesh of 3000 vertices and a deformed point cloud of 5000 points. An instance of a downstream application can be the control algorithm for a robotic hand that requires online feedback from the state of the manipulated object.
• Score: 6.003255659803736
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The world around us is full of soft objects we perceive and deform with dexterous hand movements. For a robotic hand to control soft objects, it has to acquire online state feedback of the deforming object. While RGB-D cameras can collect occluded point clouds at a rate of 30Hz, this does not represent a continuously trackable object surface. Hence, in this work, we developed a method that takes as input a template mesh which is the mesh of an object in its non-deformed state and a deformed point cloud of the same object, and then shapes the template mesh such that it matches the deformed point cloud. The reconstruction of meshes from point clouds has long been studied in the field of Computer graphics under 3D reconstruction and 4D reconstruction, however, both lack the speed and generalizability needed for robotics applications. Our model is designed using a point cloud auto-encoder and a Real-NVP architecture. Our trained model can perform mesh reconstruction and tracking at a rate of 58Hz on a template mesh of 3000 vertices and a deformed point cloud of 5000 points and is generalizable to the deformations of six different object categories which are assumed to be made of soft material in our experiments (scissors, hammer, foam brick, cleanser bottle, orange, and dice). The object meshes are taken from the YCB benchmark dataset. An instance of a downstream application can be the control algorithm for a robotic hand that requires online feedback from the state of the manipulated object which would allow online grasp adaptation in a closed-loop manner. Furthermore, the tracking capacity of our method can help in the system identification of deforming objects in a marker-free approach. In future work, we will extend our trained model to generalize beyond six object categories and additionally to real-world deforming point clouds.

Related papers

• PokeFlex: A Real-World Dataset of Deformable Objects for Robotics [17.533143584534155]
  PokeFlex is a dataset featuring real-world paired and annotated multimodal data that includes 3D textured meshes, point clouds, RGB images, and depth maps. Such data can be leveraged for several downstream tasks such as online 3D mesh reconstruction. We demonstrate a use case for the PokeFlex dataset in online 3D mesh reconstruction.
  arXiv  Detail & Related papers  (2024-10-10T07:54:17Z)
• DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects [13.138509669247508]
  Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the manipulated object and a point cloud of the goal shape. This shape embedding enables the robot to learn a visual servo controller that computes the desired robot end-effector action to
  arXiv  Detail & Related papers  (2023-05-08T04:08:06Z)
• Variational Relational Point Completion Network for Robust 3D Classification [59.80993960827833]
  Vari point cloud completion methods tend to generate global shape skeletons hence lack fine local details. This paper proposes a variational framework, point Completion Network (VRCNet) with two appealing properties. VRCNet shows great generalizability and robustness on real-world point cloud scans.
  arXiv  Detail & Related papers  (2023-04-18T17:03:20Z)
• StarNet: Style-Aware 3D Point Cloud Generation [82.30389817015877]
  StarNet is able to reconstruct and generate high-fidelity and even 3D point clouds using a mapping network. Our framework achieves comparable state-of-the-art performance on various metrics in the point cloud reconstruction and generation tasks.
  arXiv  Detail & Related papers  (2023-03-28T08:21:44Z)
• AdaPoinTr: Diverse Point Cloud Completion with Adaptive Geometry-Aware Transformers [94.11915008006483]
  We present a new method that reformulates point cloud completion as a set-to-set translation problem. We design a new model, called PoinTr, which adopts a Transformer encoder-decoder architecture for point cloud completion. Our method attains 6.53 CD on PCN, 0.81 CD on ShapeNet-55 and 0.392 MMD on real-world KITTI.
  arXiv  Detail & Related papers  (2023-01-11T16:14:12Z)
• N-Cloth: Predicting 3D Cloth Deformation with Mesh-Based Networks [69.94313958962165]
  We present a novel mesh-based learning approach (N-Cloth) for plausible 3D cloth deformation prediction. We use graph convolution to transform the cloth and object meshes into a latent space to reduce the non-linearity in the mesh space. Our approach can handle complex cloth meshes with up to $100$K triangles and scenes with various objects corresponding to SMPL humans, Non-SMPL humans, or rigid bodies.
  arXiv  Detail & Related papers  (2021-12-13T03:13:11Z)
• Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds [7.1659268120093635]
  Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the object being manipulated and a point cloud of the goal shape to learn a low-dimensional representation of the object shape.
  arXiv  Detail & Related papers  (2021-10-10T02:34:57Z)
• PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers [81.71904691925428]
  We present a new method that reformulates point cloud completion as a set-to-set translation problem. We also design a new model, called PoinTr, that adopts a transformer encoder-decoder architecture for point cloud completion. Our method outperforms state-of-the-art methods by a large margin on both the new benchmarks and the existing ones.
  arXiv  Detail & Related papers  (2021-08-19T17:58:56Z)
• DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation [5.733365759103406]
  We propose a novel approach to 3D deformable object manipulation leveraging a deep neural network called DeformerNet. We explicitly use 3D point clouds as the state representation and apply Convolutional Neural Network on point clouds to learn the 3D features. Once trained in an end-to-end fashion, DeformerNet directly maps the current point cloud of a deformable object, as well as a target point cloud shape, to the desired displacement in robot gripper position.
  arXiv  Detail & Related papers  (2021-07-16T18:20:58Z)
• Meshing Point Clouds with Predicted Intrinsic-Extrinsic Ratio Guidance [30.863194319818223]
  We propose to leverage the input point cloud as much as possible, by only adding connectivity information to existing points. Our key innovation is a surrogate of local connectivity, calculated by comparing the intrinsic/extrinsic metrics. We demonstrate that our method can not only preserve details, handle ambiguous structures, but also possess strong generalizability to unseen categories.
  arXiv  Detail & Related papers  (2020-07-17T22:36:00Z)
• ShapeAdv: Generating Shape-Aware Adversarial 3D Point Clouds [78.25501874120489]
  We develop shape-aware adversarial 3D point cloud attacks by leveraging the learned latent space of a point cloud auto-encoder. Different from prior works, the resulting adversarial 3D point clouds reflect the shape variations in the 3D point cloud space while still being close to the original one.
  arXiv  Detail & Related papers  (2020-05-24T00:03:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.