Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking

• URL: http://arxiv.org/abs/2302.09502v1
• Date: Sun, 19 Feb 2023 07:48:12 GMT
• Title: Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking
• Authors: Zixuan Huang, Xingyu Lin, David Held
• Abstract summary: We propose a self-supervised method to finetune a mesh reconstruction model in the real world. We show that we can improve the quality of the reconstructed mesh without requiring human annotations.
• Score: 18.288330275993328
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State estimation is one of the greatest challenges for cloth manipulation due to cloth's high dimensionality and self-occlusion. Prior works propose to identify the full state of crumpled clothes by training a mesh reconstruction model in simulation. However, such models are prone to suffer from a sim-to-real gap due to differences between cloth simulation and the real world. In this work, we propose a self-supervised method to finetune a mesh reconstruction model in the real world. Since the full mesh of crumpled cloth is difficult to obtain in the real world, we design a special data collection scheme and an action-conditioned model-based cloth tracking method to generate pseudo-labels for self-supervised learning. By finetuning the pretrained mesh reconstruction model on this pseudo-labeled dataset, we show that we can improve the quality of the reconstructed mesh without requiring human annotations, and improve the performance of downstream manipulation task.

Related papers

• Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering [11.228453237603834]
  We present a novel fine-grained multi-view hand mesh reconstruction method that leverages inverse rendering to restore hand poses and intricate details. We also introduce a novel Hand Albedo and Mesh (HAM) optimization module to refine both the hand mesh and textures. Our proposed approach outperforms the state-of-the-art methods on both reconstruction accuracy and rendering quality.
  arXiv  Detail & Related papers  (2024-07-08T07:28:24Z)
• A Two-stage Personalized Virtual Try-on Framework with Shape Control and Texture Guidance [7.302929117437442]
  This paper proposes a brand new personalized virtual try-on model (PE-VITON), which uses the two stages (shape control and texture guidance) to decouple the clothing attributes. The proposed model can effectively solve the problems of weak reduction of clothing folds, poor generation effect under complex human posture, blurred edges of clothing, and unclear texture styles in traditional try-on methods.
  arXiv  Detail & Related papers  (2023-12-24T13:32:55Z)
• An Adversarial Active Sampling-based Data Augmentation Framework for Manufacturable Chip Design [55.62660894625669]
  Lithography modeling is a crucial problem in chip design to ensure a chip design mask is manufacturable. Recent developments in machine learning have provided alternative solutions in replacing the time-consuming lithography simulations with deep neural networks. We propose a litho-aware data augmentation framework to resolve the dilemma of limited data and improve the machine learning model performance.
  arXiv  Detail & Related papers  (2022-10-27T20:53:39Z)
• Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation [18.288330275993328]
  Self-occlusion is challenging for cloth manipulation, as it makes it difficult to estimate the full state of the cloth. We leverage recent advances in pose estimation for cloth to build a system that uses explicit occlusion reasoning to unfold a crumpled cloth.
  arXiv  Detail & Related papers  (2022-06-06T20:15:02Z)
• SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video [48.23424267130425]
  SelfRecon recovers space-time coherent geometries from a monocular self-rotating human video. Explicit methods require a predefined template mesh for a given sequence, while the template is hard to acquire for a specific subject. Implicit methods support arbitrary topology and have high quality due to continuous geometric representation.
  arXiv  Detail & Related papers  (2022-01-30T11:49:29Z)
• Probabilistic Modeling for Human Mesh Recovery [73.11532990173441]
  This paper focuses on the problem of 3D human reconstruction from 2D evidence. We recast the problem as learning a mapping from the input to a distribution of plausible 3D poses.
  arXiv  Detail & Related papers  (2021-08-26T17:55:11Z)
• Learning Visible Connectivity Dynamics for Cloth Smoothing [17.24004979796887]
  We propose to learn a particle-based dynamics model from a partial point cloud observation. To overcome the challenges of partial observability, we infer which visible points are connected on the underlying cloth mesh. We show that our method greatly outperforms previous state-of-the-art model-based and model-free reinforcement learning methods in simulation.
  arXiv  Detail & Related papers  (2021-05-21T15:03:29Z)
• Improved Autoregressive Modeling with Distribution Smoothing [106.14646411432823]
  Autoregressive models excel at image compression, but their sample quality is often lacking. Inspired by a successful adversarial defense method, we incorporate randomized smoothing into autoregressive generative modeling.
  arXiv  Detail & Related papers  (2021-03-28T09:21:20Z)
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation [83.16948429592621]
  We propose a new learning framework that jointly optimize both the visual representation model and the dynamics model. We show substantial improvements over standard model-based learning techniques across our rope and cloth manipulation suite.
  arXiv  Detail & Related papers  (2020-03-11T17:55:15Z)
• Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animals [56.65062746564091]
  We aim to capture the pose of neuroscience model organisms, without using any manual supervision, to study how neural circuits orchestrate behaviour. Our key contribution is the explicit and independent modeling of appearance, shape and poses in an unpaired image translation framework. We demonstrate improved pose estimation accuracy on Drosophila melanogaster (fruit fly), Caenorhabditis elegans (worm) and Danio rerio (zebrafish)
  arXiv  Detail & Related papers  (2020-01-23T15:34:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.