Related papers: Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance

Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance

URL: http://arxiv.org/abs/2407.13842v2
Date: Thu, 25 Jul 2024 10:51:19 GMT
Title: Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance
Authors: Toan Nguyen, Minh Nhat Vu, Baoru Huang, An Vuong, Quan Vuong, Ngan Le, Thieu Vo, Anh Nguyen,
Abstract summary: We present a new approach for language-driven 6-DoF grasp detection in cluttered point clouds. The proposed negative prompt strategy directs the detection process toward the desired object while steering away from unwanted ones. Our method enables an end-to-end framework where humans can command the robot to grasp desired objects in a cluttered scene using natural language.
Score: 13.246380364455494
License: http://creativecommons.org/licenses/by-sa/4.0/
Abstract: 6-DoF grasp detection has been a fundamental and challenging problem in robotic vision. While previous works have focused on ensuring grasp stability, they often do not consider human intention conveyed through natural language, hindering effective collaboration between robots and users in complex 3D environments. In this paper, we present a new approach for language-driven 6-DoF grasp detection in cluttered point clouds. We first introduce Grasp-Anything-6D, a large-scale dataset for the language-driven 6-DoF grasp detection task with 1M point cloud scenes and more than 200M language-associated 3D grasp poses. We further introduce a novel diffusion model that incorporates a new negative prompt guidance learning strategy. The proposed negative prompt strategy directs the detection process toward the desired object while steering away from unwanted ones given the language input. Our method enables an end-to-end framework where humans can command the robot to grasp desired objects in a cluttered scene using natural language. Intensive experimental results show the effectiveness of our method in both benchmarking experiments and real-world scenarios, surpassing other baselines. In addition, we demonstrate the practicality of our approach in real-world robotic applications. Our project is available at https://airvlab.github.io/grasp-anything.

Related papers

Language-driven Grasp Detection with Mask-guided Attention [10.231956034184265]
We propose a new method for language-driven grasp detection with mask-guided attention. Our approach integrates visual data, segmentation mask features, and natural language instructions. Our work introduces a new framework for language-driven grasp detection, paving the way for language-driven robotic applications.
arXiv Detail & Related papers (2024-07-29T10:55:17Z)
Language-driven Grasp Detection [12.78625719116471]
We introduce a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We propose a new language-driven grasp detection method based on diffusion models. Our method outperforms state-of-the-art approaches and allows real-world robotic grasping.
arXiv Detail & Related papers (2024-06-13T16:06:59Z)
MENTOR: Multilingual tExt detectioN TOward leaRning by analogy [59.37382045577384]
We propose a framework to detect and identify both seen and unseen language regions inside scene images. "MENTOR" is the first work to realize a learning strategy between zero-shot learning and few-shot learning for multilingual scene text detection.
arXiv Detail & Related papers (2024-03-12T03:35:17Z)
Language-Guided 3D Object Detection in Point Cloud for Autonomous Driving [91.91552963872596]
We propose a new multi-modal visual grounding task, termed LiDAR Grounding. It jointly learns the LiDAR-based object detector with the language features and predicts the targeted region directly from the detector. Our work offers a deeper insight into the LiDAR-based grounding task and we expect it presents a promising direction for the autonomous driving community.
arXiv Detail & Related papers (2023-05-25T06:22:10Z)
Language-Driven Representation Learning for Robotics [115.93273609767145]
Recent work in visual representation learning for robotics demonstrates the viability of learning from large video datasets of humans performing everyday tasks. We introduce a framework for language-driven representation learning from human videos and captions. We find that Voltron's language-driven learning outperform the prior-of-the-art, especially on targeted problems requiring higher-level control.
arXiv Detail & Related papers (2023-02-24T17:29:31Z)
Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding [42.04502185508723]
We propose a new large Language-guided SHape grAsPing datasEt to promote 3D part-level affordance and grasping ability learning. From the perspective of robotic cognition, we design a two-stage fine-grained robotic grasping framework (named LangPartGPD) Our method combines the advantages of human-robot collaboration and large language models (LLMs) Results show our method achieves competitive performance in 3D geometry fine-grained grounding, object affordance inference, and 3D part-aware grasping tasks.
arXiv Detail & Related papers (2023-01-27T07:00:54Z)
Learning 6-DoF Object Poses to Grasp Category-level Objects by Language Instructions [74.63313641583602]
This paper studies the task of any objects grasping from the known categories by free-form language instructions. We bring these disciplines together on this open challenge, which is essential to human-robot interaction. We propose a language-guided 6-DoF category-level object localization model to achieve robotic grasping by comprehending human intention.
arXiv Detail & Related papers (2022-05-09T04:25:14Z)
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation [80.29069988090912]
We study the problem of learning a range of vision-based manipulation tasks from a large offline dataset of robot interaction. We propose to leverage offline robot datasets with crowd-sourced natural language labels. We find that our approach outperforms both goal-image specifications and language conditioned imitation techniques by more than 25%.
arXiv Detail & Related papers (2021-09-02T17:42:13Z)
INVIGORATE: Interactive Visual Grounding and Grasping in Clutter [56.00554240240515]
INVIGORATE is a robot system that interacts with human through natural language and grasps a specified object in clutter. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. We build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules.
arXiv Detail & Related papers (2021-08-25T07:35:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.