TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objects

• URL: http://arxiv.org/abs/2603.13869v1
• Date: Sat, 14 Mar 2026 09:59:34 GMT
• Title: TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objects
• Abstract summary: Dexterous manipulation enables complex tasks but suffers from self-occlusion, severe depth noise, and depth information loss when manipulating transparent objects.<n>This paper proposes TransDex, a 3D visuo-tactile fusion motor policy based on point cloud reconstruction pre-training.<n>Results from transparent object manipulation experiments conducted on a real robotic system demonstrate that TransDex outperforms existing baseline methods.
• Score: 14.591201169062158
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dexterous manipulation enables complex tasks but suffers from self-occlusion, severe depth noise, and depth information loss when manipulating transparent objects. To solve this problem, this paper proposes TransDex, a 3D visuo-tactile fusion motor policy based on point cloud reconstruction pre-training. Specifically, we first propose a self-supervised point cloud reconstruction pre-training approach based on Transformer. This method accurately recovers the 3D structure of objects from interactive point clouds of dexterous hands, even when random noise and large-scale masking are added. Building on this, TransDex is constructed in which perceptual encoding adopts a fine-grained hierarchical scheme and multi-round attention mechanisms adaptively fuse features of the robotic arm and dexterous hand to enable differentiated motion prediction. Results from transparent object manipulation experiments conducted on a real robotic system demonstrate that TransDex outperforms existing baseline methods. Further analysis validates the generalization capabilities of TransDex and the effectiveness of its individual components.

Related papers

• Visual Geometry Transformer in the Wild: Distractor-Free 3D Reconstruction [77.75911117889915]
  Current end-to-end multi-view 3D reconstruction methods achieve impressive results, but rely on a restrictive static assumption.<n>This reliance on idealized inputs causes even the most advanced methods to fail in real-world settings.<n>We propose Visual Geometry Transformer in the Wild, an end-to-end framework for robust reconstruction from inconsistent views.
  arXiv  Detail & Related papers  (2026-06-22T02:52:22Z)
• Trans2Occ: Voxel Occupancy Estimation and Grasp for Transparent Objects from Simulation to Reality [125.76063573012853]
  We present a framework for transparent object perception and manipulation based on single-view RGB input.<n>Our approach predicts voxel-space occupancy directly from a single image, providing a geometry-aware representation that supports downstream robotic grasping.
  arXiv  Detail & Related papers  (2026-06-01T06:59:13Z)
• ComPose: When to Trust Hands for Object Pose Tracking [44.085148707189035]
  ComPose is a 6DoF object tracking framework designed for hand-aware object pose estimation from RGB video.<n>Our method harmonizes hand motions as a textitcomplementary cue for object tracking.<n>Result can effectively transfer to downstream robot manipulation by enabling robots to reconstruct human actions from online videos.
  arXiv  Detail & Related papers  (2026-05-22T11:39:49Z)
• Structural Action Transformer for 3D Dexterous Manipulation [80.07649565189035]
  Cross-embodiment skill transfer is a challenge for high-DoF robotic hands.<n>Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations.<n>This paper proposes a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective.
  arXiv  Detail & Related papers  (2026-03-04T11:38:12Z)
• TransBridge: Boost 3D Object Detection by Scene-Level Completion with Transformer Decoder [66.22997415145467]
  This paper presents a joint completion and detection framework that improves the detection feature in sparse areas.<n> Specifically, we propose TransBridge, a novel transformer-based up-sampling block that fuses the features from the detection and completion networks.<n>The results show that our framework consistently improves end-to-end 3D object detection, with the mean average precision (mAP) ranging from 0.7 to 1.5 across multiple methods.
  arXiv  Detail & Related papers  (2025-12-12T00:08:03Z)
• ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation? [23.73409728939164]
  This paper introduces ManipGPT, a framework designed to predict optimal interaction areas for articulated objects.<n>We create a dataset of 9.9k simulated and real images to bridge the visual sim-to-real gap.<n>We significantly improve part-level affordance segmentation, adapting the model's in-context segmentation capabilities to robot manipulation scenarios.
  arXiv  Detail & Related papers  (2024-12-13T11:22:01Z)
• ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation [18.140839442955485]
  We develop a vision transformer-based algorithm for stereo depth recovery of transparent objects.<n>Our method incorporates a parameter-aligned, domain-adaptive, and physically realistic Sim2Real simulation for efficient data generation.<n>Our experimental results demonstrate the model's exceptional Sim2Real generalizability in real-world scenarios.
  arXiv  Detail & Related papers  (2024-09-13T15:44:38Z)
• 3D Face Modeling via Weakly-supervised Disentanglement Network joint Identity-consistency Prior [62.80458034704989]
  Generative 3D face models featuring disentangled controlling factors hold immense potential for diverse applications in computer vision and computer graphics. Previous 3D face modeling methods face a challenge as they demand specific labels to effectively disentangle these factors. This paper introduces a Weakly-Supervised Disentanglement Framework, denoted as WSDF, to facilitate the training of controllable 3D face models without an overly stringent labeling requirement.
  arXiv  Detail & Related papers  (2024-04-25T11:50:47Z)
• RFTrans: Leveraging Refractive Flow of Transparent Objects for Surface Normal Estimation and Manipulation [50.10282876199739]
  This paper introduces RFTrans, an RGB-D-based method for surface normal estimation and manipulation of transparent objects. It integrates the RFNet, which predicts refractive flow, object mask, and boundaries, followed by the F2Net, which estimates surface normal from the refractive flow. A real-world robot grasping task witnesses an 83% success rate, proving that refractive flow can help enable direct sim-to-real transfer.
  arXiv  Detail & Related papers  (2023-11-21T07:19:47Z)
• Transformers Solve the Limited Receptive Field for Monocular Depth Prediction [82.90445525977904]
  We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers. This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
  arXiv  Detail & Related papers  (2021-03-22T18:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.