End-to-End Dexterous Grasp Learning from Single-View Point Clouds via a Multi-Object Scene Dataset

• URL: http://arxiv.org/abs/2603.15410v1
• Date: Mon, 16 Mar 2026 15:21:13 GMT
• Title: End-to-End Dexterous Grasp Learning from Single-View Point Clouds via a Multi-Object Scene Dataset
• Abstract summary: Current mainstream grasping datasets predominantly focus on single-object scenarios and predefined grasp configurations.<n>We propose DGS-Net, an end-to-end grasp prediction network capable of learning dense grasp configurations from single-view point clouds in multi-object scene.<n>Our dataset comprises 307 objects, 240 multi-object scenes, and over 350k validated grasps.
• Score: 36.127347599544514
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dexterous grasping in multi-object scene constitutes a fundamental challenge in robotic manipulation. Current mainstream grasping datasets predominantly focus on single-object scenarios and predefined grasp configurations, often neglecting environmental interference and the modeling of dexterous pre-grasp gesture, thereby limiting their generalizability in real-world applications. To address this, we propose DGS-Net, an end-to-end grasp prediction network capable of learning dense grasp configurations from single-view point clouds in multi-object scene. Furthermore, we propose a two-stage grasp data generation strategy that progresses from dense single-object grasp synthesis to dense scene-level grasp generation. Our dataset comprises 307 objects, 240 multi-object scenes, and over 350k validated grasps. By explicitly modeling grasp offsets and pre-grasp configurations, the dataset provides more robust and accurate supervision for dexterous grasp learning. Experimental results show that DGS-Net achieves grasp success rates of 88.63\% in simulation and 78.98\% on a real robotic platform, while exhibiting lower penetration with a mean penetration depth of 0.375 mm and penetration volume of 559.45 mm^3, outperforming existing methods and demonstrating strong effectiveness and generalization capability. Our dataset is available at https://github.com/4taotao8/DGS-Net.

Related papers

• GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation [3.906714757786654]
  GOAG is a novel deep generative model that learns a compact latent representation of a specific gripper's contact surface distribution.<n>We validate our approach through extensive experiments on established grasp protocols in both simulated and real-world scenarios.<n>Our method delivers state-of-the-art results on the objects from the MultiDex dataset, achieving an average success rate of 86.93%.
  arXiv  Detail & Related papers  (2026-08-20T08:03:39Z)
• SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments [17.165500070631442]
  Vision-language models (VLMs) offer a natural way to specify requirements using language.<n>Existing approaches either use a VLM to predict the grasp directly with limited spatial awareness, or train the VLM together with the grasping model.<n>We propose SeededGrasp, a novel data-efficient framework that enables a VLM to predict a seed point to be used as conditioning for a subsequent lightweight grasp-generation model.
  arXiv  Detail & Related papers  (2026-07-22T14:27:45Z)
• Decoupling Semantics and Geometric Grounding: Spatial Visual Prompts for Language-Conditioned Imitation Learning [21.67233368353003]
  We propose SVP-IL, a decoupled architecture that explicitly extracts spatial visual grounding from the action generation loop.<n>In experiments, SVP-IL significantly outperforms state-of-the-art VLAs and pure visuomotor baselines.
  arXiv  Detail & Related papers  (2026-06-24T03:45:06Z)
• URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model [76.08429266631823]
  We propose an end-to-end automatic reconstruction framework based on a 3D multimodal large language model (MLLM)<n>URDF-Anything utilizes an autoregressive prediction framework based on point-cloud and text multimodal input to jointly optimize geometric segmentation and kinematic parameter prediction.<n> Experiments on both simulated and real-world datasets demonstrate that our method significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-11-02T13:45:51Z)
• DexVLG: Dexterous Vision-Language-Grasp Model at Scale [59.5613919093295]
  There is little research on functional grasping with large models for human-like dexterous hands.<n>We introduce DexVLG, a large Vision-Language-Grasp model for Dexterous grasp pose prediction aligned with language instructions.<n>We generate a dataset of 170 million dexterous grasp poses mapped to semantic parts across 174,000 objects in simulation, paired with detailed part-level captions.
  arXiv  Detail & Related papers  (2025-07-03T16:05:25Z)
• GraspClutter6D: A Large-scale Real-world Dataset for Robust Perception and Grasping in Cluttered Scenes [5.289647064481469]
  We present GraspClutter6D, a large-scale real-world grasping dataset featuring 1,000 cluttered scenes with dense arrangements.<n>We benchmark state-of-the-art segmentation, object pose estimation, and grasp detection methods to provide key insights into challenges in cluttered environments.<n>We validate the dataset's effectiveness as a training resource, demonstrating that grasping networks trained on GraspClutter6D significantly outperform those trained on existing datasets in both simulation and real-world experiments.
  arXiv  Detail & Related papers  (2025-04-09T13:15:46Z)
• Advancing ALS Applications with Large-Scale Pre-training: Dataset Development and Downstream Assessment [6.606615641354963]
  The pre-training and fine-tuning paradigm has revolutionized satellite remote sensing applications.<n>We construct a large-scale ALS point cloud dataset and evaluate its impact on downstream applications.<n>Our results show that the pre-trained models significantly outperform their scratch counterparts across all downstream tasks.
  arXiv  Detail & Related papers  (2025-01-09T09:21:09Z)
• SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes [58.89295356901823]
  Self-supervised monocular depth estimation has shown impressive results in static scenes. It relies on the multi-view consistency assumption for training networks, however, that is violated in dynamic object regions. We introduce an external pretrained monocular depth estimation model for generating single-image depth prior. Our model can predict sharp and accurate depth maps, even when training from monocular videos of highly-dynamic scenes.
  arXiv  Detail & Related papers  (2022-11-07T16:17:47Z)
• MetaGraspNet: A Large-Scale Benchmark Dataset for Vision-driven Robotic Grasping via Physics-based Metaverse Synthesis [78.26022688167133]
  We present a large-scale benchmark dataset for vision-driven robotic grasping via physics-based metaverse synthesis. The proposed dataset contains 100,000 images and 25 different object types. We also propose a new layout-weighted performance metric alongside the dataset for evaluating object detection and segmentation performance.
  arXiv  Detail & Related papers  (2021-12-29T17:23:24Z)
• Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans [103.92680099373567]
  This paper introduces a pipeline to parametrically sample and render multi-task vision datasets from comprehensive 3D scans from the real world. Changing the sampling parameters allows one to "steer" the generated datasets to emphasize specific information. Common architectures trained on a generated starter dataset reached state-of-the-art performance on multiple common vision tasks and benchmarks.
  arXiv  Detail & Related papers  (2021-10-11T04:21:46Z)
• Deep Learning based Segmentation of Fish in Noisy Forward Looking MBES Images [1.5469452301122177]
  We build on recent advances in Deep Learning (DL) and Convolutional Neural Networks (CNNs) for semantic segmentation. We demonstrate an end-to-end approach for a fish/non-fish probability prediction for all range-azimuth positions projected by an imaging sonar. We show that our model proves the desired performance and has learned to harness the importance of semantic context.
  arXiv  Detail & Related papers  (2020-06-16T09:57:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.