Viewpoint-Agnostic Grasp Pipeline using VLM and Partial Observations

• URL: http://arxiv.org/abs/2603.07866v1
• Date: Mon, 09 Mar 2026 00:42:32 GMT
• Title: Viewpoint-Agnostic Grasp Pipeline using VLM and Partial Observations
• Abstract summary: We present an end-to-end pipeline for language-guided grasping that bridges open-vocabulary target selection to safe grasp execution on a real robot.<n>Given a natural-language command, the system grounds the target in RGB using open-vocabulary detection and promptable instance segmentation.<n>We then generate and collision-filter 6-DoF grasp candidates and select an executable grasp using safety-oriented grasps that account for reachability, approach feasibility, and clearance.
• Score: 0.6263680699548958
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robust grasping in cluttered, unstructured environments remains challenging for mobile legged manipulators due to occlusions that lead to partial observations, unreliable depth estimates, and the need for collision-free, execution-feasible approaches. In this paper we present an end-to-end pipeline for language-guided grasping that bridges open-vocabulary target selection to safe grasp execution on a real robot. Given a natural-language command, the system grounds the target in RGB using open-vocabulary detection and promptable instance segmentation, extracts an object-centric point cloud from RGB-D, and improves geometric reliability under occlusion via back-projected depth compensation and two-stage point cloud completion. We then generate and collision-filter 6-DoF grasp candidates and select an executable grasp using safety-oriented heuristics that account for reachability, approach feasibility, and clearance. We evaluate the method on a quadruped robot with an arm in two cluttered tabletop scenarios, using paired trials against a view-dependent baseline. The proposed approach achieves a 90% overall success rate (9/10) against 30% (3/10) for the baseline, demonstrating substantially improved robustness to occlusions and partial observations in clutter.

Related papers

• Back to the Feature: Zero-Shot 6DoF Pose Estimation via Dense Local Features [0.0]
  We present B2ose, a training-free zero-shot method for 6DoF pose estimation of unseen objects from RGB images.<n>B2ose extracts dense patch-level features that generalize across the synthetic-to-real domain gap without task-specific fine-tuning.<n>On the seven core datasets of the BOP Benchmark, B2ose achieves 40.7 mean AR without refinement and 56.4 refinement.
  arXiv  Detail & Related papers  (2026-09-06T16:57:27Z)
• Partial Skeleton Visibility for Action Recognition: A Constrained Field-of-View Approach [60.596944437968524]
  PartialVisGraph is a novel framework for robust skeleton-based action recognition under constrained field-of-view.<n>It consistently achieves state-of-the-art accuracy under partial visibility, with gains of up to 68.8% on subsets with severe FoV restrictions.<n>Our approach offers a principled and practical pathway toward deployable skeleton-based action understanding in unconstrained environments.
  arXiv  Detail & Related papers  (2026-07-01T10:03:11Z)
• Active Adversarial Perturbation-driven Associative Memory Retrieval for RGB-Event Visual Object Tracking [58.338848435042536]
  RGB-Event tracking improves localization by fusing RGB appearance textures and dense temporal motion cues from event sensors.<n>Real-world scenes suffer diverse structured signal degradations that hinder traditional multi-modal fusion.<n>We present a hierarchical perturbation and retrieval framework tailored for RGB-Event tracking with robustness against partial target missing and modal degradation.
  arXiv  Detail & Related papers  (2026-06-24T23:34:06Z)
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning [99.49739831030388]
  We study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning.<n>We propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface.
  arXiv  Detail & Related papers  (2026-06-11T17:59:36Z)
• A Contrastive Fewshot RGBD Traversability Segmentation Framework for Indoor Robotic Navigation [0.7258309433541204]
  Pure vision-based models often fail to detect thin obstacles, such as chair legs, which can pose serious safety risks.<n>We propose a multi-modal segmentation framework that leverages RGB images and sparse 1D laser depth information.<n>Our method achieves up to 9% higher mIoU under both 1-shot and 5-shot settings.
  arXiv  Detail & Related papers  (2026-03-06T22:56:39Z)
• Learning Fine-Grained Correspondence with Cross-Perspective Perception for Open-Vocabulary 6D Object Pose Estimation [14.262846967061947]
  Fine-grained Correspondence Pose Estimation (FiCoP) is a framework that transitions from noise-prone global matching to spatially-constrained patch-level correspondence.<n>FiCoP improves Average Recall by 8.0% and 6.1%, respectively, compared to the state-of-the-art method.
  arXiv  Detail & Related papers  (2026-01-20T03:48:54Z)
• Into the Unknown: Towards using Generative Models for Sampling Priors of Environment Uncertainty for Planning in Configuration Spaces [28.37021202108478]
  Priors are vital for planning under partial observability, yet difficult to obtain in practice.<n>We present a probabilistic-based pipeline that leverages large-scale pretrained generative models to produce priors in a zero-shot manner.<n>We establish a Matterport3D benchmark of rooms partially visible through doorways, where a robot must navigate to an unobserved target object.
  arXiv  Detail & Related papers  (2025-10-13T05:08:48Z)
• VOGUE: Guiding Exploration with Visual Uncertainty Improves Multimodal Reasoning [62.09195763860549]
  Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models (LLMs) but struggles with exploration.<n>We introduce $textbfVOGUE (Visual Uncertainty Guided Exploration)$, a novel method that shifts exploration from the output (text) to the input (visual) space.<n>Our work shows that grounding exploration in the inherent uncertainty of visual inputs is an effective strategy for improving multimodal reasoning.
  arXiv  Detail & Related papers  (2025-10-01T20:32:08Z)
• CurbNet: Curb Detection Framework Based on LiDAR Point Cloud Segmentation [8.502741852406904]
  This paper introduces CurbNet, a novel framework for curb detection utilizing point cloud segmentation.<n>We have developed the 3D-Curb dataset based on Semantic KITTI, currently the largest and most diverse collection of curb point clouds.<n>To tackle the challenges posed by the uneven distribution of curb features on the xy-plane and their dependence on high-frequency features along the z-axis, we introduce the Multi-Scale and Channel Attention (MSCA) module.
  arXiv  Detail & Related papers  (2024-03-25T14:13:09Z)
• RGB-based Category-level Object Pose Estimation via Decoupled Metric Scale Recovery [72.13154206106259]
  We propose a novel pipeline that decouples the 6D pose and size estimation to mitigate the influence of imperfect scales on rigid transformations. Specifically, we leverage a pre-trained monocular estimator to extract local geometric information. A separate branch is designed to directly recover the metric scale of the object based on category-level statistics.
  arXiv  Detail & Related papers  (2023-09-19T02:20:26Z)
• ContrastMotion: Self-supervised Scene Motion Learning for Large-Scale LiDAR Point Clouds [21.6511040107249]
  We propose a novel self-supervised motion estimator for LiDAR-based autonomous driving via BEV representation. We predict scene motion via feature-level consistency between pillars in consecutive frames, which can eliminate the effect caused by noise points and view-changing point clouds in dynamic scenes.
  arXiv  Detail & Related papers  (2023-04-25T05:46:24Z)
• Self-Supervised Arbitrary-Scale Point Clouds Upsampling via Implicit Neural Representation [79.60988242843437]
  We propose a novel approach that achieves self-supervised and magnification-flexible point clouds upsampling simultaneously. Experimental results demonstrate that our self-supervised learning based scheme achieves competitive or even better performance than supervised learning based state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-04-18T07:18:25Z)
• Occlusion-Robust Object Pose Estimation with Holistic Representation [42.27081423489484]
  State-of-the-art (SOTA) object pose estimators take a two-stage approach. We develop a novel occlude-and-blackout batch augmentation technique. We also develop a multi-precision supervision architecture to encourage holistic pose representation learning.
  arXiv  Detail & Related papers  (2021-10-22T08:00:26Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)
• Scope Head for Accurate Localization in Object Detection [135.9979405835606]
  We propose a novel detector coined as ScopeNet, which models anchors of each location as a mutually dependent relationship. With our concise and effective design, the proposed ScopeNet achieves state-of-the-art results on COCO.
  arXiv  Detail & Related papers  (2020-05-11T04:00:09Z)
• Robust 6D Object Pose Estimation by Learning RGB-D Features [59.580366107770764]
  We propose a novel discrete-continuous formulation for rotation regression to resolve this local-optimum problem. We uniformly sample rotation anchors in SO(3), and predict a constrained deviation from each anchor to the target, as well as uncertainty scores for selecting the best prediction. Experiments on two benchmarks: LINEMOD and YCB-Video, show that the proposed method outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-02-29T06:24:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.