DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy

• URL: http://arxiv.org/abs/2505.11032v2
• Date: Mon, 19 May 2025 07:28:03 GMT
• Title: DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy
• Authors: Yuran Wang, Ruihai Wu, Yue Chen, Jiarui Wang, Jiaqi Liang, Ziyu Zhu, Haoran Geng, Jitendra Malik, Pieter Abbeel, Hao Dong,
• Abstract summary: Garment manipulation is a critical challenge due to the diversity in garment categories, geometries, and deformations.<n>We propose DexGarmentLab, the first environment specifically designed for dexterous (especially bimanual) garment manipulation.<n>It features large-scale high-quality 3D assets for 15 task scenarios, and refines simulation techniques tailored for garment modeling to reduce the sim-to-real gap.
• Score: 74.9519138296936
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Garment manipulation is a critical challenge due to the diversity in garment categories, geometries, and deformations. Despite this, humans can effortlessly handle garments, thanks to the dexterity of our hands. However, existing research in the field has struggled to replicate this level of dexterity, primarily hindered by the lack of realistic simulations of dexterous garment manipulation. Therefore, we propose DexGarmentLab, the first environment specifically designed for dexterous (especially bimanual) garment manipulation, which features large-scale high-quality 3D assets for 15 task scenarios, and refines simulation techniques tailored for garment modeling to reduce the sim-to-real gap. Previous data collection typically relies on teleoperation or training expert reinforcement learning (RL) policies, which are labor-intensive and inefficient. In this paper, we leverage garment structural correspondence to automatically generate a dataset with diverse trajectories using only a single expert demonstration, significantly reducing manual intervention. However, even extensive demonstrations cannot cover the infinite states of garments, which necessitates the exploration of new algorithms. To improve generalization across diverse garment shapes and deformations, we propose a Hierarchical gArment-manipuLation pOlicy (HALO). It first identifies transferable affordance points to accurately locate the manipulation area, then generates generalizable trajectories to complete the task. Through extensive experiments and detailed analysis of our method and baseline, we demonstrate that HALO consistently outperforms existing methods, successfully generalizing to previously unseen instances even with significant variations in shape and deformation where others fail. Our project page is available at: https://wayrise.github.io/DexGarmentLab/.

Related papers

• GraphGarment: Learning Garment Dynamics for Bimanual Cloth Manipulation Tasks [7.4467523788133585]
  GraphGarment is a novel approach that models garment dynamics based on robot control inputs.<n>We use graphs to represent the interactions between the robot end-effector and the garment.<n>We conduct four experiments using six types of garments to validate our approach in both simulation and real-world settings.
  arXiv  Detail & Related papers  (2025-03-04T17:35:48Z)
• GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation [12.940189262612677]
  GarmentLab is a content-rich benchmark and realistic simulation designed for deformable object and garment manipulation.<n>Our benchmark encompasses a diverse range of garment types, robotic systems and manipulators.<n>We evaluate state-of-the-art vision methods, reinforcement learning, and imitation learning approaches on these tasks.
  arXiv  Detail & Related papers  (2024-11-02T10:09:08Z)
• SKT: Integrating State-Aware Keypoint Trajectories with Vision-Language Models for Robotic Garment Manipulation [82.61572106180705]
  This paper presents a unified approach using vision-language models (VLMs) to improve keypoint prediction across various garment categories. We created a large-scale synthetic dataset using advanced simulation techniques, allowing scalable training without extensive real-world data. Experimental results indicate that the VLM-based method significantly enhances keypoint detection accuracy and task success rates.
  arXiv  Detail & Related papers  (2024-09-26T17:26:16Z)
• AniDress: Animatable Loose-Dressed Avatar from Sparse Views Using Garment Rigging Model [58.035758145894846]
  We introduce AniDress, a novel method for generating animatable human avatars in loose clothes using very sparse multi-view videos. A pose-driven deformable neural radiance field conditioned on both body and garment motions is introduced, providing explicit control of both parts. Our method is able to render natural garment dynamics that deviate highly from the body and well to generalize to both unseen views and poses.
  arXiv  Detail & Related papers  (2024-01-27T08:48:18Z)
• UniFolding: Towards Sample-efficient, Scalable, and Generalizable Robotic Garment Folding [53.38503172679482]
  UniFolding is a sample-efficient, scalable, and generalizable robotic system for unfolding and folding garments. UniFolding employs the proposed UFONet neural network to integrate unfolding and folding decisions into a single policy model. The system is tested on two garment types: long-sleeve and short-sleeve shirts.
  arXiv  Detail & Related papers  (2023-11-02T14:25:10Z)
• SwinGar: Spectrum-Inspired Neural Dynamic Deformation for Free-Swinging Garments [6.821050909555717]
  We present a spectrum-inspired learning-based approach for generating clothing deformations with dynamic effects and personalized details. Our proposed method overcomes limitations by providing a unified framework that predicts dynamic behavior for different garments. We develop a dynamic clothing deformation estimator that integrates frequency-controllable attention mechanisms with long short-term memory.
  arXiv  Detail & Related papers  (2023-08-05T09:09:50Z)
• HOOD: Hierarchical Graphs for Generalized Modelling of Clothing Dynamics [84.29846699151288]
  Our method is agnostic to body shape and applies to tight-fitting garments as well as loose, free-flowing clothing. As one key contribution, we propose a hierarchical message-passing scheme that efficiently propagates stiff stretching modes.
  arXiv  Detail & Related papers  (2022-12-14T14:24:00Z)
• Learning Fabric Manipulation in the Real World with Human Videos [10.608723220309678]
  Fabric manipulation is a long-standing challenge in robotics due to the enormous state space and complex dynamics. Most prior methods rely heavily on simulation, which is still limited by the large sim-to-real gap of deformable objects. A promising alternative is to learn fabric manipulation directly from watching humans perform the task.
  arXiv  Detail & Related papers  (2022-11-05T07:09:15Z)
• Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN [66.3650689395967]
  We propose a texture-preserving end-to-end network, the PAtch-routed SpaTially-Adaptive GAN (PASTA-GAN), that facilitates real-world unpaired virtual try-on. To disentangle the style and spatial information of each garment, PASTA-GAN consists of an innovative patch-routed disentanglement module.
  arXiv  Detail & Related papers  (2021-11-20T08:36:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.