Clothes Grasping and Unfolding Based on RGB-D Semantic Segmentation

• URL: http://arxiv.org/abs/2305.03259v2
• Date: Mon, 8 May 2023 12:44:30 GMT
• Title: Clothes Grasping and Unfolding Based on RGB-D Semantic Segmentation
• Authors: Xingyu Zhu, Xin Wang, Jonathan Freer, Hyung Jin Chang, Yixing Gao
• Abstract summary: We propose a novel Bi-directional Fractal Cross Fusion Network (BiFCNet) for semantic segmentation. We use RGB images with rich color features as input to our network in which the Fractal Cross Fusion module fuses RGB and depth data. To reduce the cost of real data collection, we propose a data augmentation method based on an adversarial strategy.
• Score: 21.950751953721817
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Clothes grasping and unfolding is a core step in robotic-assisted dressing. Most existing works leverage depth images of clothes to train a deep learning-based model to recognize suitable grasping points. These methods often utilize physics engines to synthesize depth images to reduce the cost of real labeled data collection. However, the natural domain gap between synthetic and real images often leads to poor performance of these methods on real data. Furthermore, these approaches often struggle in scenarios where grasping points are occluded by the clothing item itself. To address the above challenges, we propose a novel Bi-directional Fractal Cross Fusion Network (BiFCNet) for semantic segmentation, enabling recognition of graspable regions in order to provide more possibilities for grasping. Instead of using depth images only, we also utilize RGB images with rich color features as input to our network in which the Fractal Cross Fusion (FCF) module fuses RGB and depth data by considering global complex features based on fractal geometry. To reduce the cost of real data collection, we further propose a data augmentation method based on an adversarial strategy, in which the color and geometric transformations simultaneously process RGB and depth data while maintaining the label correspondence. Finally, we present a pipeline for clothes grasping and unfolding from the perspective of semantic segmentation, through the addition of a strategy for grasp point selection from segmentation regions based on clothing flatness measures, while taking into account the grasping direction. We evaluate our BiFCNet on the public dataset NYUDv2 and obtained comparable performance to current state-of-the-art models. We also deploy our model on a Baxter robot, running extensive grasping and unfolding experiments as part of our ablation studies, achieving an 84% success rate.

Related papers

• Diffusion-based RGB-D Semantic Segmentation with Deformable Attention Transformer [10.982521876026281]
  We introduce a diffusion-based framework to address the RGB-D semantic segmentation problem. We demonstrate that utilizing a Deformable Attention Transformer as the encoder to extract features from depth images effectively captures the characteristics of invalid regions in depth measurements.
  arXiv  Detail & Related papers  (2024-09-23T15:23:01Z)
• Semantic RGB-D Image Synthesis [22.137419841504908]
  We introduce semantic RGB-D image synthesis to address this problem. Current approaches, however, are uni-modal and cannot cope with multi-modal data. We propose a generator for multi-modal data that separates modal-independent information of the semantic layout from the modal-dependent information.
  arXiv  Detail & Related papers  (2023-08-22T11:16:24Z)
• Attentive Multimodal Fusion for Optical and Scene Flow [24.08052492109655]
  Existing methods typically rely solely on RGB images or fuse the modalities at later stages. We propose a novel deep neural network approach named FusionRAFT, which enables early-stage information fusion between sensor modalities. Our approach exhibits improved robustness in the presence of noise and low-lighting conditions that affect the RGB images.
  arXiv  Detail & Related papers  (2023-07-28T04:36:07Z)
• Semantic keypoint-based pose estimation from single RGB frames [64.80395521735463]
  We present an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. We show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios.
  arXiv  Detail & Related papers  (2022-04-12T15:03:51Z)
• Scale Invariant Semantic Segmentation with RGB-D Fusion [12.650574326251023]
  We propose a neural network architecture for scale-invariant semantic segmentation using RGB-D images. We incorporate depth information to the RGB data for pixel-wise semantic segmentation to address the different scale objects in an outdoor scene. Our model is compact and can be easily applied to the other RGB model.
  arXiv  Detail & Related papers  (2022-04-10T12:54:27Z)
• Grasp-Oriented Fine-grained Cloth Segmentation without Real Supervision [66.56535902642085]
  This paper tackles the problem of fine-grained region detection in deformed clothes using only a depth image. We define up to 6 semantic regions of varying extent, including edges on the neckline, sleeve cuffs, and hem, plus top and bottom grasping points. We introduce a U-net based network to segment and label these parts. We show that training our network solely with synthetic data and the proposed DA yields results competitive with models trained on real data.
  arXiv  Detail & Related papers  (2021-10-06T16:31:20Z)
• Self-Supervised Representation Learning for RGB-D Salient Object Detection [93.17479956795862]
  We use Self-Supervised Representation Learning to design two pretext tasks: the cross-modal auto-encoder and the depth-contour estimation. Our pretext tasks require only a few and un RGB-D datasets to perform pre-training, which make the network capture rich semantic contexts. For the inherent problem of cross-modal fusion in RGB-D SOD, we propose a multi-path fusion module.
  arXiv  Detail & Related papers  (2021-01-29T09:16:06Z)
• Adaptive Context-Aware Multi-Modal Network for Depth Completion [107.15344488719322]
  We propose to adopt the graph propagation to capture the observed spatial contexts. We then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks.
  arXiv  Detail & Related papers  (2020-08-25T06:00:06Z)
• Progressively Guided Alternate Refinement Network for RGB-D Salient Object Detection [63.18846475183332]
  We aim to develop an efficient and compact deep network for RGB-D salient object detection. We propose a progressively guided alternate refinement network to refine it. Our model outperforms existing state-of-the-art approaches by a large margin.
  arXiv  Detail & Related papers  (2020-08-17T02:55:06Z)
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation [59.94819184452694]
  Depth information has proven to be a useful cue in the semantic segmentation of RGBD images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion. In this paper, we propose a unified and efficient Crossmodality Guided to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively.
  arXiv  Detail & Related papers  (2020-07-17T18:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.