MVTrans: Multi-View Perception of Transparent Objects

• URL: http://arxiv.org/abs/2302.11683v1
• Date: Wed, 22 Feb 2023 22:45:28 GMT
• Title: MVTrans: Multi-View Perception of Transparent Objects
• Authors: Yi Ru Wang, Yuchi Zhao, Haoping Xu, Saggi Eppel, Alan Aspuru-Guzik, Florian Shkurti, Animesh Garg
• Abstract summary: We forgo the unreliable depth map from RGB-D sensors and extend the stereo based method. Our proposed method, MVTrans, is an end-to-end multi-view architecture with multiple perception capabilities. We establish a novel procedural photo-realistic dataset generation pipeline and create a large-scale transparent object detection dataset.
• Score: 29.851395075937255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transparent object perception is a crucial skill for applications such as robot manipulation in household and laboratory settings. Existing methods utilize RGB-D or stereo inputs to handle a subset of perception tasks including depth and pose estimation. However, transparent object perception remains to be an open problem. In this paper, we forgo the unreliable depth map from RGB-D sensors and extend the stereo based method. Our proposed method, MVTrans, is an end-to-end multi-view architecture with multiple perception capabilities, including depth estimation, segmentation, and pose estimation. Additionally, we establish a novel procedural photo-realistic dataset generation pipeline and create a large-scale transparent object detection dataset, Syn-TODD, which is suitable for training networks with all three modalities, RGB-D, stereo and multi-view RGB. Project Site: https://ac-rad.github.io/MVTrans/

Related papers

• ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation [18.140839442955485]
  We develop a vision transformer-based algorithm for stereo depth recovery of transparent objects. Our method incorporates a parameter-aligned, domain-adaptive, and physically realistic Sim2Real simulation for efficient data generation. Our experimental results demonstrate the model's exceptional Sim2Real generalizability in real-world scenarios.
  arXiv  Detail & Related papers  (2024-09-13T15:44:38Z)
• ASGrasp: Generalizable Transparent Object Reconstruction and Grasping from RGB-D Active Stereo Camera [9.212504138203222]
  We propose ASGrasp, a 6-DoF grasp detection network that uses an RGB-D active stereo camera. Our system distinguishes itself by its ability to directly utilize raw IR and RGB images for transparent object geometry reconstruction. Our experiments demonstrate that ASGrasp can achieve over 90% success rate for generalizable transparent object grasping.
  arXiv  Detail & Related papers  (2024-05-09T09:44:51Z)
• Grasping the Inconspicuous [15.274311118568715]
  We study deep learning 6D pose estimation from RGB images only for transparent object grasping. Experiments demonstrate the effectiveness of RGB image space for grasping transparent objects.
  arXiv  Detail & Related papers  (2022-11-15T14:45:50Z)
• Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision [51.385731364529306]
  This paper focuses on perceiving and navigating 3D environments using echoes and RGB image. In particular, we perform depth estimation by fusing RGB image with echoes, received from multiple orientations. We show that the echoes provide holistic and in-expensive information about the 3D structures complementing the RGB image.
  arXiv  Detail & Related papers  (2022-07-03T22:31:47Z)
• Dual Swin-Transformer based Mutual Interactive Network for RGB-D Salient Object Detection [67.33924278729903]
  In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the long-range dependencies in visual inputs. Comprehensive experiments on five standard RGB-D SOD benchmark datasets demonstrate the superiority of the proposed DTMINet method.
  arXiv  Detail & Related papers  (2022-06-07T08:35:41Z)
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction [91.43066633305662]
  We propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD) Specifically, we unify three complementary tasks: depth estimation, salient object detection and contour estimation. The multi-task mechanism promotes the model to learn the task-aware features from the auxiliary tasks. Experiments show that it not only significantly surpasses the depth-based RGB-D SOD methods on multiple datasets, but also precisely predicts a high-quality depth map and salient contour at the same time.
  arXiv  Detail & Related papers  (2022-03-09T17:20:18Z)
• Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects [16.714074893209713]
  TranspareNet is a joint point cloud and depth completion method. It can complete the depth of transparent objects in cluttered and complex scenes. TranspareNet outperforms existing state-of-the-art depth completion methods on multiple datasets.
  arXiv  Detail & Related papers  (2021-09-30T21:09:09Z)
• RGB-D Saliency Detection via Cascaded Mutual Information Minimization [122.8879596830581]
  Existing RGB-D saliency detection models do not explicitly encourage RGB and depth to achieve effective multi-modal learning. We introduce a novel multi-stage cascaded learning framework via mutual information minimization to "explicitly" model the multi-modal information between RGB image and depth data.
  arXiv  Detail & Related papers  (2021-09-15T12:31:27Z)
• Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection [78.47767202232298]
  We propose a novel Cross-modality Discrepant Interaction Network (CDINet) for RGB-D SOD. Two components are designed to implement the effective cross-modality interaction. Our network outperforms $15$ state-of-the-art methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2021-08-04T11:24:42Z)
• RGB-D Local Implicit Function for Depth Completion of Transparent Objects [43.238923881620494]
  Majority of perception methods in robotics require depth information provided by RGB-D cameras. Standard 3D sensors fail to capture depth of transparent objects due to refraction and absorption of light. We present a novel framework that can complete missing depth given noisy RGB-D input.
  arXiv  Detail & Related papers  (2021-04-01T17:00:04Z)
• Is Depth Really Necessary for Salient Object Detection? [50.10888549190576]
  We make the first attempt in realizing an unified depth-aware framework with only RGB information as input for inference. Not only surpasses the state-of-the-art performances on five public RGB SOD benchmarks, but also surpasses the RGBD-based methods on five benchmarks by a large margin.
  arXiv  Detail & Related papers  (2020-05-30T13:40:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.