RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing

• URL: http://arxiv.org/abs/2407.01418v1
• Date: Mon, 1 Jul 2024 16:08:37 GMT
• Title: RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing
• Authors: Bo Ai, Stephen Tian, Haochen Shi, Yixuan Wang, Cheston Tan, Yunzhu Li, Jiajun Wu,
• Abstract summary: We introduce an approach that combines visual and tactile sensing for robotic manipulation by learning a neural, tactile-informed dynamics model. Our proposed framework, RoboPack, employs a recurrent graph neural network to estimate object states. We demonstrate our approach on a real robot equipped with a compliant Soft-Bubble tactile sensor on non-prehensile manipulation and dense packing tasks.
• Score: 38.97168020979433
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tactile feedback is critical for understanding the dynamics of both rigid and deformable objects in many manipulation tasks, such as non-prehensile manipulation and dense packing. We introduce an approach that combines visual and tactile sensing for robotic manipulation by learning a neural, tactile-informed dynamics model. Our proposed framework, RoboPack, employs a recurrent graph neural network to estimate object states, including particles and object-level latent physics information, from historical visuo-tactile observations and to perform future state predictions. Our tactile-informed dynamics model, learned from real-world data, can solve downstream robotics tasks with model-predictive control. We demonstrate our approach on a real robot equipped with a compliant Soft-Bubble tactile sensor on non-prehensile manipulation and dense packing tasks, where the robot must infer the physics properties of objects from direct and indirect interactions. Trained on only an average of 30 minutes of real-world interaction data per task, our model can perform online adaptation and make touch-informed predictions. Through extensive evaluations in both long-horizon dynamics prediction and real-world manipulation, our method demonstrates superior effectiveness compared to previous learning-based and physics-based simulation systems.

Related papers

• Physics-Encoded Graph Neural Networks for Deformation Prediction under Contact [87.69278096528156]
  In robotics, it's crucial to understand object deformation during tactile interactions. We introduce a method using Physics-Encoded Graph Neural Networks (GNNs) for such predictions. We've made our code and dataset public to advance research in robotic simulation and grasping.
  arXiv  Detail & Related papers  (2024-02-05T19:21:52Z)
• Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods [14.780597545674157]
  We investigate the effects of visual pre-training strategies on robot manipulation tasks from three fundamental perspectives. We propose a visual pre-training scheme for robot manipulation termed Vi-PRoM, which combines self-supervised learning and supervised learning.
  arXiv  Detail & Related papers  (2023-08-07T14:24:52Z)
• Combining Vision and Tactile Sensation for Video Prediction [0.0]
  We investigate the impact of integrating tactile feedback into video prediction models for physical robot interactions. We introduce two new datasets of robot pushing that use a magnetic-based tactile sensor for unsupervised learning. Our results demonstrate that incorporating tactile feedback into video prediction models improves scene prediction accuracy and enhances the agent's perception of physical interactions.
  arXiv  Detail & Related papers  (2023-04-21T18:02:15Z)
• Physics-Inspired Temporal Learning of Quadrotor Dynamics for Accurate Model Predictive Trajectory Tracking [76.27433308688592]
  Accurately modeling quadrotor's system dynamics is critical for guaranteeing agile, safe, and stable navigation. We present a novel Physics-Inspired Temporal Convolutional Network (PI-TCN) approach to learning quadrotor's system dynamics purely from robot experience. Our approach combines the expressive power of sparse temporal convolutions and dense feed-forward connections to make accurate system predictions.
  arXiv  Detail & Related papers  (2022-06-07T13:51:35Z)
• RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks [32.00371492516123]
  We present a model-based planning framework for modeling and manipulating elasto-plastic objects. Our system, RoboCraft, learns a particle-based dynamics model using graph neural networks (GNNs) to capture the structure of the underlying system. We show through experiments that with just 10 minutes of real-world robotic interaction data, our robot can learn a dynamics model that can be used to synthesize control signals to deform elasto-plastic objects into various target shapes.
  arXiv  Detail & Related papers  (2022-05-05T20:28:15Z)
• Dynamic Modeling of Hand-Object Interactions via Tactile Sensing [133.52375730875696]
  In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing.
  arXiv  Detail & Related papers  (2021-09-09T16:04:14Z)
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling [86.9726984929758]
  We focus on the integration of incomplete physics models into deep generative models. We propose a VAE architecture in which a part of the latent space is grounded by physics. We demonstrate generative performance improvements over a set of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-02-25T20:28:52Z)
• Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interaction [24.72947291987545]
  Key challenge for an agent learning to interact with the world is to reason about physical properties of objects. We propose a novel approach for modeling the dynamics of a robot's interactions directly from unlabeled 3D point clouds and images.
  arXiv  Detail & Related papers  (2020-08-02T11:04:49Z)
• Learning Predictive Models From Observation and Interaction [137.77887825854768]
  Learning predictive models from interaction with the world allows an agent, such as a robot, to learn about how the world works. However, learning a model that captures the dynamics of complex skills represents a major challenge. We propose a method to augment the training set with observational data of other agents, such as humans.
  arXiv  Detail & Related papers  (2019-12-30T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.