Language-driven Grasp Detection

• URL: http://arxiv.org/abs/2406.09489v1
• Date: Thu, 13 Jun 2024 16:06:59 GMT
• Title: Language-driven Grasp Detection
• Authors: An Dinh Vuong, Minh Nhat Vu, Baoru Huang, Nghia Nguyen, Hieu Le, Thieu Vo, Anh Nguyen,
• Abstract summary: We introduce a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We propose a new language-driven grasp detection method based on diffusion models. Our method outperforms state-of-the-art approaches and allows real-world robotic grasping.
• Score: 12.78625719116471
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Grasp detection is a persistent and intricate challenge with various industrial applications. Recently, many methods and datasets have been proposed to tackle the grasp detection problem. However, most of them do not consider using natural language as a condition to detect the grasp poses. In this paper, we introduce Grasp-Anything++, a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We utilize foundation models to create a large-scale scene corpus with corresponding images and grasp prompts. We approach the language-driven grasp detection task as a conditional generation problem. Drawing on the success of diffusion models in generative tasks and given that language plays a vital role in this task, we propose a new language-driven grasp detection method based on diffusion models. Our key contribution is the contrastive training objective, which explicitly contributes to the denoising process to detect the grasp pose given the language instructions. We illustrate that our approach is theoretically supportive. The intensive experiments show that our method outperforms state-of-the-art approaches and allows real-world robotic grasping. Finally, we demonstrate our large-scale dataset enables zero-short grasp detection and is a challenging benchmark for future work. Project website: https://airvlab.github.io/grasp-anything/

Related papers

• Language-driven Grasp Detection with Mask-guided Attention [10.231956034184265]
  We propose a new method for language-driven grasp detection with mask-guided attention. Our approach integrates visual data, segmentation mask features, and natural language instructions. Our work introduces a new framework for language-driven grasp detection, paving the way for language-driven robotic applications.
  arXiv  Detail & Related papers  (2024-07-29T10:55:17Z)
• Lightweight Language-driven Grasp Detection using Conditional Consistency Model [10.254392362201308]
  We present a new approach for language-driven grasp detection that leverages the concept of lightweight diffusion models. Our method can effectively encode visual and textual information, enabling more accurate and versatile grasp positioning. We further validate our method in real-world robotic experiments to demonstrate its fast inference time capability.
  arXiv  Detail & Related papers  (2024-07-25T11:39:20Z)
• Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance [13.246380364455494]
  We present a new approach for language-driven 6-DoF grasp detection in cluttered point clouds. The proposed negative prompt strategy directs the detection process toward the desired object while steering away from unwanted ones. Our method enables an end-to-end framework where humans can command the robot to grasp desired objects in a cluttered scene using natural language.
  arXiv  Detail & Related papers  (2024-07-18T18:24:51Z)
• Graspness Discovery in Clutters for Fast and Accurate Grasp Detection [57.81325062171676]
  "graspness" is a quality based on geometry cues that distinguishes graspable areas in cluttered scenes. We develop a neural network named cascaded graspness model to approximate the searching process. Experiments on a large-scale benchmark, GraspNet-1Billion, show that our method outperforms previous arts by a large margin.
  arXiv  Detail & Related papers  (2024-06-17T02:06:47Z)
• POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models [62.23255433487586]
  We propose an unsupervised fine-tuning framework to fine-tune the model or prompt on the unlabeled target data. We demonstrate how to apply our method to both language-augmented vision and masked-language models by aligning the discrete distributions extracted from the prompts and target data.
  arXiv  Detail & Related papers  (2023-04-29T22:05:22Z)
• Ensemble Transfer Learning for Multilingual Coreference Resolution [60.409789753164944]
  A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. We design a simple but effective ensemble-based framework that combines various transfer learning techniques. We also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.
  arXiv  Detail & Related papers  (2023-01-22T18:22:55Z)
• Sample Efficient Approaches for Idiomaticity Detection [6.481818246474555]
  This work explores sample efficient methods of idiomaticity detection. In particular, we study the impact of Pattern Exploit Training (PET), a few-shot method of classification, and BERTRAM, an efficient method of creating contextual embeddings. Our experiments show that whilePET improves performance on English, they are much less effective on Portuguese and Galician, leading to an overall performance about on par with vanilla mBERT.
  arXiv  Detail & Related papers  (2022-05-23T13:46:35Z)
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages [87.5457337866383]
  We introduce the Image-Grounded Language Understanding Evaluation benchmark. IGLUE brings together visual question answering, cross-modal retrieval, grounded reasoning, and grounded entailment tasks across 20 diverse languages. We find that translate-test transfer is superior to zero-shot transfer and that few-shot learning is hard to harness for many tasks.
  arXiv  Detail & Related papers  (2022-01-27T18:53:22Z)
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models [86.02610674750345]
  Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. We apply 14 adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy.
  arXiv  Detail & Related papers  (2021-11-04T12:59:55Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Training [32.800766653254634]
  We present the most comprehensive study of cross-lingual stance detection to date. We use 15 diverse datasets in 12 languages from 6 language families. For our experiments, we build on pattern-exploiting training, proposing the addition of a novel label encoder.
  arXiv  Detail & Related papers  (2021-09-13T15:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.