Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics

• URL: http://arxiv.org/abs/2003.12698v2
• Date: Thu, 12 Nov 2020 01:01:59 GMT
• Title: Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics
• Authors: Aditya Ganapathi, Priya Sundaresan, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita, Jennifer Grannen, Minho Hwang, Ryan Hoque, Joseph E. Gonzalez, Nawid Jamali, Katsu Yamane, Soshi Iba, Ken Goldberg
• Abstract summary: We learn visual correspondences for deformable fabrics across different configurations in simulation. The learned correspondences can be used to compute geometrically equivalent actions in a new fabric configuration. Results also suggest to fabrics of various colors, sizes, and shapes.
• Score: 35.84249614544505
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robotic fabric manipulation is challenging due to the infinite dimensional configuration space, self-occlusion, and complex dynamics of fabrics. There has been significant prior work on learning policies for specific deformable manipulation tasks, but comparatively less focus on algorithms which can efficiently learn many different tasks. In this paper, we learn visual correspondences for deformable fabrics across different configurations in simulation and show that this representation can be used to design policies for a variety of tasks. Given a single demonstration of a new task from an initial fabric configuration, the learned correspondences can be used to compute geometrically equivalent actions in a new fabric configuration. This makes it possible to robustly imitate a broad set of multi-step fabric smoothing and folding tasks on multiple physical robotic systems. The resulting policies achieve 80.3% average task success rate across 10 fabric manipulation tasks on two different robotic systems, the da Vinci surgical robot and the ABB YuMi. Results also suggest robustness to fabrics of various colors, sizes, and shapes. See https://tinyurl.com/fabric-descriptors for supplementary material and videos.

Related papers

• UniFolding: Towards Sample-efficient, Scalable, and Generalizable Robotic Garment Folding [53.38503172679482]
  UniFolding is a sample-efficient, scalable, and generalizable robotic system for unfolding and folding garments. UniFolding employs the proposed UFONet neural network to integrate unfolding and folding decisions into a single policy model. The system is tested on two garment types: long-sleeve and short-sleeve shirts.
  arXiv  Detail & Related papers  (2023-11-02T14:25:10Z)
• Towards Multi-Layered 3D Garments Animation [135.77656965678196]
  Existing approaches mostly focus on single-layered garments driven by only human bodies and struggle to handle general scenarios. We propose a novel data-driven method, called LayersNet, to model garment-level animations as particle-wise interactions in a micro physics system. Our experiments show that LayersNet achieves superior performance both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-05-17T17:53:04Z)
• Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention [4.2940878152791555]
  We present a novel multi-step cloth manipulation planning framework named Foldformer. We experimentally evaluate Foldsformer on four representative sequential multi-step manipulation tasks. Our approach can be transferred from simulation to the real world without additional training or domain randomization.
  arXiv  Detail & Related papers  (2023-01-08T09:15:45Z)
• Learning Fabric Manipulation in the Real World with Human Videos [10.608723220309678]
  Fabric manipulation is a long-standing challenge in robotics due to the enormous state space and complex dynamics. Most prior methods rely heavily on simulation, which is still limited by the large sim-to-real gap of deformable objects. A promising alternative is to learn fabric manipulation directly from watching humans perform the task.
  arXiv  Detail & Related papers  (2022-11-05T07:09:15Z)
• Learning Tool Morphology for Contact-Rich Manipulation Tasks with Differentiable Simulation [27.462052737553055]
  We present an end-to-end framework to automatically learn tool morphology for contact-rich manipulation tasks by leveraging differentiable physics simulators. In our approach, we instead only need to define the objective with respect to the task performance and enable learning a robust morphology by randomizing the task variations. We demonstrate the effectiveness of our method for designing new tools in several scenarios such as winding ropes, flipping a box and pushing peas onto a scoop in simulation.
  arXiv  Detail & Related papers  (2022-11-04T00:57:36Z)
• MetaGraspNet: A Large-Scale Benchmark Dataset for Vision-driven Robotic Grasping via Physics-based Metaverse Synthesis [78.26022688167133]
  We present a large-scale benchmark dataset for vision-driven robotic grasping via physics-based metaverse synthesis. The proposed dataset contains 100,000 images and 25 different object types. We also propose a new layout-weighted performance metric alongside the dataset for evaluating object detection and segmentation performance.
  arXiv  Detail & Related papers  (2021-12-29T17:23:24Z)
• A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation [63.1610540170754]
  We focus on the problem of visual non-prehensile planar manipulation. We propose a novel architecture that combines video decoding neural models with priors from contact mechanics. We find that our modular and fully differentiable architecture performs better than learning-only methods on unseen objects and motions.
  arXiv  Detail & Related papers  (2021-11-09T18:39:45Z)
• PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics [89.81550748680245]
  We introduce a new differentiable physics benchmark called PasticineLab. In each task, the agent uses manipulators to deform the plasticine into the desired configuration. We evaluate several existing reinforcement learning (RL) methods and gradient-based methods on this benchmark.
  arXiv  Detail & Related papers  (2021-04-07T17:59:23Z)
• VisuoSpatial Foresight for Physical Sequential Fabric Manipulation [22.008305401551418]
  We build upon the Visual Foresight framework to learn fabric dynamics that can be efficiently reused to accomplish different sequential fabric manipulation tasks. In this work, we vary 4 components of VSF, including data generation, the choice of visual dynamics model, cost function, and optimization procedure. Results suggest that training visual dynamics models using longer, corner-based actions can improve the efficiency of fabric folding by 76% and enable a physical sequential fabric folding task that VSF could not previously perform with 90% reliability.
  arXiv  Detail & Related papers  (2021-02-19T06:06:49Z)
• VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation [24.262746504997683]
  We extend the Visual Foresight framework to learn fabric dynamics that can be efficiently reused to accomplish different fabric manipulation tasks. We experimentally evaluate VSF on multi-step fabric smoothing and folding tasks against 5 baseline methods in simulation and on the da Vinci Research Kit (dVRK) surgical robot without any demonstrations at train or test time.
  arXiv  Detail & Related papers  (2020-03-19T23:12:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.