Target-Oriented Object Grasping via Multimodal Human Guidance

• URL: http://arxiv.org/abs/2408.11138v1
• Date: Tue, 20 Aug 2024 18:42:56 GMT
• Title: Target-Oriented Object Grasping via Multimodal Human Guidance
• Abstract summary: Traditional grasp detection methods analyze the entire scene to predict grasps, leading to redundancy and inefficiency. In this work, we reconsider 6-DoF grasp detection from a target-referenced perspective and propose a Target-Oriented Grasp Network (TOGNet) TOGNet specifically targets local, object-agnostic region patches to predict grasps more efficiently.
• Score: 6.031799490542892
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the context of human-robot interaction and collaboration scenarios, robotic grasping still encounters numerous challenges. Traditional grasp detection methods generally analyze the entire scene to predict grasps, leading to redundancy and inefficiency. In this work, we reconsider 6-DoF grasp detection from a target-referenced perspective and propose a Target-Oriented Grasp Network (TOGNet). TOGNet specifically targets local, object-agnostic region patches to predict grasps more efficiently. It integrates seamlessly with multimodal human guidance, including language instructions, pointing gestures, and interactive clicks. Thus our system comprises two primary functional modules: a guidance module that identifies the target object in 3D space and TOGNet, which detects region-focal 6-DoF grasps around the target, facilitating subsequent motion planning. Through 50 target-grasping simulation experiments in cluttered scenes, our system achieves a success rate improvement of about 13.7%. In real-world experiments, we demonstrate that our method excels in various target-oriented grasping scenarios.

Related papers

• GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation [3.906714757786654]
  GOAG is a novel deep generative model that learns a compact latent representation of a specific gripper's contact surface distribution.<n>We validate our approach through extensive experiments on established grasp protocols in both simulated and real-world scenarios.<n>Our method delivers state-of-the-art results on the objects from the MultiDex dataset, achieving an average success rate of 86.93%.
  arXiv  Detail & Related papers  (2026-08-20T08:03:39Z)
• STEM: Semantic Target Search and Exploration using MAVs in Cluttered Environments [17.009653719709213]
  This paper proposes a novel framework that utilizes a semantically-guided viewpoint planner to minimize target search and exploration time.<n>Specifically, we develop a planner that generates efficient semantic exploration plans by prioritizing viewpoints that likely lead to the target.
  arXiv  Detail & Related papers  (2026-05-30T15:00:58Z)
• Zero-Shot Stance Detection in the Wild: Dynamic Target Generation and Multi-Target Adaptation [13.424443287821118]
  We propose a novel task: zero-shot stance detection in the wild with Dynamic Target Generation and Multi-Target Adaptation.<n>We construct a Chinese social media stance detection dataset and design multi-dimensional evaluation metrics.
  arXiv  Detail & Related papers  (2026-01-27T17:04:18Z)
• VPTracker: Global Vision-Language Tracking via Visual Prompt and MLLM [45.56517073754981]
  Vision-Language Tracking aims to continuously localize objects described by a visual template and a language description.<n>Existing methods, however, are typically limited to local search, making them prone to failures under viewpoint changes.<n>We introduce the first global tracking framework based on Multimodal Large Language Models (VPTracker), exploiting their powerful semantic reasoning to locate targets.
  arXiv  Detail & Related papers  (2025-12-28T06:12:28Z)
• Sight Over Site: Perception-Aware Reinforcement Learning for Efficient Robotic Inspection [57.37596278863949]
  In this work, we revisit inspection from a perception-aware perspective.<n>We propose an end-to-end reinforcement learning framework that explicitly incorporates target visibility as the primary objective.<n>We show that our method outperforms existing classical and learning-based navigation approaches.
  arXiv  Detail & Related papers  (2025-09-22T15:14:02Z)
• Evaluating Pointing Gestures for Target Selection in Human-Robot Collaboration [0.0]
  This study introduces a method for localizing pointed targets within a planar workspace.<n>The approach employs pose estimation, and a simple geometric model based on shoulder-wrist extension to extract gesturing data from an RGB-D stream.
  arXiv  Detail & Related papers  (2025-06-27T10:51:31Z)
• DINO-CoDT: Multi-class Collaborative Detection and Tracking with Vision Foundation Models [11.34839442803445]
  We propose a multi-class collaborative detection and tracking framework tailored for diverse road users.<n>We first present a detector with a global spatial attention fusion (GSAF) module, enhancing multi-scale feature learning for objects of varying sizes.<n>Next, we introduce a tracklet RE-IDentification (REID) module that leverages visual semantics with a vision foundation model to effectively reduce ID SWitch (IDSW) errors.
  arXiv  Detail & Related papers  (2025-06-09T02:49:10Z)
• Tell Me What to Track: Infusing Robust Language Guidance for Enhanced Referring Multi-Object Tracking [10.614327633823462]
  Referring multi-object tracking (RMOT) is an emerging cross-modal task that aims to localize an arbitrary number of targets. We conduct a collaborative matching strategy to alleviate the impact of the imbalance, boosting the ability to detect newborn targets. In the encoder, we integrate and enhance the cross-modal and multi-scale fusion, overcoming the bottlenecks in previous work.
  arXiv  Detail & Related papers  (2024-12-17T05:43:35Z)
• Local Occupancy-Enhanced Object Grasping with Multiple Triplanar Projection [24.00828999360765]
  This paper addresses the challenge of robotic grasping of general objects. The proposed model first runs by proposing a number of most likely grasp points in the scene. Around each grasp point, a module is designed to infer any voxel in its neighborhood to be either void or occupied by some object. The model further estimates 6-DoF grasp poses utilizing the local occupancy-enhanced object shape information.
  arXiv  Detail & Related papers  (2024-07-22T16:22:28Z)
• Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance [13.246380364455494]
  We present a new approach for language-driven 6-DoF grasp detection in cluttered point clouds. The proposed negative prompt strategy directs the detection process toward the desired object while steering away from unwanted ones. Our method enables an end-to-end framework where humans can command the robot to grasp desired objects in a cluttered scene using natural language.
  arXiv  Detail & Related papers  (2024-07-18T18:24:51Z)
• Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge [62.981429762309226]
  The ability to actively ground task instructions from an egocentric view is crucial for AI agents to accomplish tasks or assist humans virtually. We propose to improve phrase grounding models' ability on localizing the active objects by: learning the role of objects undergoing change and extracting them accurately from the instructions. We evaluate our framework on Ego4D and Epic-Kitchens datasets.
  arXiv  Detail & Related papers  (2023-10-23T16:14:05Z)
• Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach [9.029861710944704]
  We propose a robot learning approach to interact with novel objects and collect each object's training label. The Singulation-and-Grasping (SaG) policy is trained through end-to-end reinforcement learning. Our system achieves 70% singulation success rate in simulated cluttered scenes.
  arXiv  Detail & Related papers  (2022-07-19T15:01:36Z)
• Object Manipulation via Visual Target Localization [64.05939029132394]
  Training agents to manipulate objects, poses many challenges. We propose an approach that explores the environment in search for target objects, computes their 3D coordinates once they are located, and then continues to estimate their 3D locations even when the objects are not visible. Our evaluations show a massive 3x improvement in success rate over a model that has access to the same sensory suite.
  arXiv  Detail & Related papers  (2022-03-15T17:59:01Z)
• INVIGORATE: Interactive Visual Grounding and Grasping in Clutter [56.00554240240515]
  INVIGORATE is a robot system that interacts with human through natural language and grasps a specified object in clutter. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. We build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules.
  arXiv  Detail & Related papers  (2021-08-25T07:35:21Z)
• Tracking by Joint Local and Global Search: A Target-aware Attention based Approach [63.50045332644818]
  We propose a novel target-aware attention mechanism (termed TANet) to conduct joint local and global search for robust tracking. Specifically, we extract the features of target object patch and continuous video frames, then we track and feed them into a decoder network to generate target-aware global attention maps. In the tracking procedure, we integrate the target-aware attention with multiple trackers by exploring candidate search regions for robust tracking.
  arXiv  Detail & Related papers  (2021-06-09T06:54:15Z)
• PDNet: Towards Better One-stage Object Detection with Prediction Decoupling [37.83405509385431]
  We propose a prediction-target-decoupled detector named PDNet to establish a more flexible detection paradigm. With a single ResNeXt-64x4d-101 as the backbone, our detector achieves 48.7 AP with single-scale testing.
  arXiv  Detail & Related papers  (2021-04-28T16:48:04Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• Dynamic Attention guided Multi-Trajectory Analysis for Single Object Tracking [62.13213518417047]
  We propose to introduce more dynamics by devising a dynamic attention-guided multi-trajectory tracking strategy. In particular, we construct dynamic appearance model that contains multiple target templates, each of which provides its own attention for locating the target in the new frame. After spanning the whole sequence, we introduce a multi-trajectory selection network to find the best trajectory that delivers improved tracking performance.
  arXiv  Detail & Related papers  (2021-03-30T05:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.