SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation

• URL: http://arxiv.org/abs/2309.08728v1
• Date: Fri, 15 Sep 2023 19:27:44 GMT
• Title: SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation
• Authors: Alison Bartsch, Charlotte Avra, Amir Barati Farimani
• Abstract summary: State representation for materials that exhibit plastic behavior, like modeling clay or bread dough, is difficult because they permanently deform under stress and are constantly changing shape. We propose a system that uses point clouds as the state representation and leverages pre-trained point cloud reconstruction Transformer to learn a latent dynamics model to predict material deformations given a grasp action.
• Score: 8.517406772939292
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deformable object manipulation presents a unique set of challenges in robotic manipulation by exhibiting high degrees of freedom and severe self-occlusion. State representation for materials that exhibit plastic behavior, like modeling clay or bread dough, is also difficult because they permanently deform under stress and are constantly changing shape. In this work, we investigate each of these challenges using the task of robotic sculpting with a parallel gripper. We propose a system that uses point clouds as the state representation and leverages pre-trained point cloud reconstruction Transformer to learn a latent dynamics model to predict material deformations given a grasp action. We design a novel action sampling algorithm that reasons about geometrical differences between point clouds to further improve the efficiency of model-based planners. All data and experiments are conducted entirely in the real world. Our experiments show the proposed system is able to successfully capture the dynamics of clay, and is able to create a variety of simple shapes.

Related papers

• RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing [38.97168020979433]
  We introduce an approach that combines visual and tactile sensing for robotic manipulation by learning a neural, tactile-informed dynamics model. Our proposed framework, RoboPack, employs a recurrent graph neural network to estimate object states. We demonstrate our approach on a real robot equipped with a compliant Soft-Bubble tactile sensor on non-prehensile manipulation and dense packing tasks.
  arXiv  Detail & Related papers  (2024-07-01T16:08:37Z)
• REACTO: Reconstructing Articulated Objects from a Single Video [64.89760223391573]
  We propose a novel deformation model that enhances the rigidity of each part while maintaining flexible deformation of the joints. Our method outperforms previous works in producing higher-fidelity 3D reconstructions of general articulated objects.
  arXiv  Detail & Related papers  (2024-04-17T08:01:55Z)
• DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics [97.75188532559952]
  We propose a principled framework that abstracts dexterous manipulation skills from human demonstration. We then train a skill model using demonstrations for planning over action abstractions in imagination. To evaluate the effectiveness of our approach, we introduce a suite of six challenging dexterous deformable object manipulation tasks.
  arXiv  Detail & Related papers  (2023-03-27T17:59:49Z)
• RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks [32.00371492516123]
  We present a model-based planning framework for modeling and manipulating elasto-plastic objects. Our system, RoboCraft, learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes.
  arXiv  Detail & Related papers  (2022-05-05T20:28:15Z)
• ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation [135.10594078615952]
  We introduce ACID, an action-conditional visual dynamics model for volumetric deformable objects. A benchmark contains over 17,000 action trajectories with six types of plush toys and 78 variants. Our model achieves the best performance in geometry, correspondence, and dynamics predictions.
  arXiv  Detail & Related papers  (2022-03-14T04:56:55Z)
• LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies [78.17425779503047]
  We propose a novel neural implicit representation for the human body. It is fully differentiable and optimizable with disentangled shape and pose latent spaces. Our model can be trained and fine-tuned directly on non-watertight raw data with well-designed losses.
  arXiv  Detail & Related papers  (2021-11-30T04:10:57Z)
• Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds [7.1659268120093635]
  Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape control rely on hand-crafted features to represent the object shape and require training of object-specific control models. We overcome these issues through the use of our novel DeformerNet neural network architecture, which operates on a partial-view point cloud of the object being manipulated and a point cloud of the goal shape to learn a low-dimensional representation of the object shape.
  arXiv  Detail & Related papers  (2021-10-10T02:34:57Z)
• gradSim: Differentiable simulation for system identification and visuomotor control [66.37288629125996]
  We present gradSim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering. Our unified graph enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision.
  arXiv  Detail & Related papers  (2021-04-06T16:32:01Z)
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation [83.16948429592621]
  We propose a new learning framework that jointly optimize both the visual representation model and the dynamics model. We show substantial improvements over standard model-based learning techniques across our rope and cloth manipulation suite.
  arXiv  Detail & Related papers  (2020-03-11T17:55:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.