Differentiable Physics Simulation of Dynamics-Augmented Neural Objects

• URL: http://arxiv.org/abs/2210.09420v2
• Date: Thu, 20 Oct 2022 04:51:53 GMT
• Title: Differentiable Physics Simulation of Dynamics-Augmented Neural Objects
• Authors: Simon Le Cleac'h, Hong-Xing Yu, Michelle Guo, Taylor A. Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, Mac Schwager
• Abstract summary: We present a differentiable pipeline for simulating the motion of objects that represent their geometry as a continuous density field parameterized as a deep network. We estimate the dynamical properties of the object, including its mass, center of mass, and inertia matrix. This allows a robot to autonomously build object models that are visually and dynamically accurate from still images and videos of objects in motion.
• Score: 40.587385809005355
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a differentiable pipeline for simulating the motion of objects that represent their geometry as a continuous density field parameterized as a deep network. This includes Neural Radiance Fields (NeRFs), and other related models. From the density field, we estimate the dynamical properties of the object, including its mass, center of mass, and inertia matrix. We then introduce a differentiable contact model based on the density field for computing normal and friction forces resulting from collisions. This allows a robot to autonomously build object models that are visually and dynamically accurate from still images and videos of objects in motion. The resulting Dynamics-Augmented Neural Objects (DANOs) are simulated with an existing differentiable simulation engine, Dojo, interacting with other standard simulation objects, such as spheres, planes, and robots specified as URDFs. A robot can use this simulation to optimize grasps and manipulation trajectories of neural objects, or to improve the neural object models through gradient-based real-to-simulation transfer. We demonstrate the pipeline to learn the coefficient of friction of a bar of soap from a real video of the soap sliding on a table. We also learn the coefficient of friction and mass of a Stanford bunny through interactions with a Panda robot arm from synthetic data, and we optimize trajectories in simulation for the Panda arm to push the bunny to a goal location.

Related papers

• Automated 3D Physical Simulation of Open-world Scene with Gaussian Splatting [22.40115216094332]
  We present Sim Anything, a physics-based approach that endows static 3D objects with interactive dynamics. Inspired by human visual reasoning, we propose MLLM-based Physical Property Perception. We also simulate objects in an open-world scene with particles sampled via the Physical-Geometric Adaptive Sampling.
  arXiv  Detail & Related papers  (2024-11-19T12:52:21Z)
• Physics-Encoded Graph Neural Networks for Deformation Prediction under Contact [87.69278096528156]
  In robotics, it's crucial to understand object deformation during tactile interactions. We introduce a method using Physics-Encoded Graph Neural Networks (GNNs) for such predictions. We've made our code and dataset public to advance research in robotic simulation and grasping.
  arXiv  Detail & Related papers  (2024-02-05T19:21:52Z)
• DiffuseBot: Breeding Soft Robots With Physics-Augmented Generative Diffusion Models [102.13968267347553]
  We present DiffuseBot, a physics-augmented diffusion model that generates soft robot morphologies capable of excelling in a wide spectrum of tasks. We showcase a range of simulated and fabricated robots along with their capabilities.
  arXiv  Detail & Related papers  (2023-11-28T18:58:48Z)
• DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation [81.11585774044848]
  We present DeepSimHO, a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization.
  arXiv  Detail & Related papers  (2023-10-11T05:34:36Z)
• Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos [8.012771454339353]
  We propose a novel approach for real-to-sim which tracks rigid objects in 3D from RGB-D images and infers physical properties of the objects. We demonstrate and evaluate our approach on a real-world dataset.
  arXiv  Detail & Related papers  (2023-09-27T14:46:01Z)
• RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks [32.00371492516123]
  We present a model-based planning framework for modeling and manipulating elasto-plastic objects. Our system, RoboCraft, learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes.
  arXiv  Detail & Related papers  (2022-05-05T20:28:15Z)
• Virtual Elastic Objects [18.228492027143307]
  We build virtual objects that behave like their real-world counterparts, even when subject to novel interactions. We use a differentiable, particle-based simulator to use deformation fields to find representative material parameters. We present our results using a dataset of 12 objects under a variety of force fields, which will be shared with the community.
  arXiv  Detail & Related papers  (2022-01-12T18:59:03Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)
• GeoSim: Photorealistic Image Simulation with Geometry-Aware Composition [81.24107630746508]
  We present GeoSim, a geometry-aware image composition process that synthesizes novel urban driving scenes. We first build a diverse bank of 3D objects with both realistic geometry and appearance from sensor data. The resulting synthetic images are photorealistic, traffic-aware, and geometrically consistent, allowing image simulation to scale to complex use cases.
  arXiv  Detail & Related papers  (2021-01-16T23:00:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.