Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning

• URL: http://arxiv.org/abs/2604.08726v1
• Date: Thu, 09 Apr 2026 19:39:45 GMT
• Title: Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning
• Abstract summary: Bimanual manipulation requires reasoning about where to interact with an object and which arm should perform each action.<n>Existing approaches either treat affordance prediction as coarse part segmentation or rely on geometrics for arm assignment.<n>We propose a hierarchical framework for task-aware bimanual affordance prediction.
• Score: 22.1291953459429
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bimanual manipulation requires reasoning about where to interact with an object and which arm should perform each action, a joint affordance localization and arm allocation problem that geometry-only planners cannot resolve without semantic understanding of task intent. Existing approaches either treat affordance prediction as coarse part segmentation or rely on geometric heuristics for arm assignment, failing to jointly reason about task-relevant contact regions and arm allocation. We reframe bimanual manipulation as a joint affordance localization and arm allocation problem and propose a hierarchical framework for task-aware bimanual affordance prediction that leverages a Vision-Language Model (VLM) to generalize across object categories and task descriptions without requiring category-specific training. Our approach fuses multi-view RGB-D observations into a consistent 3D scene representation and generates global 6-DoF grasp candidates, which are then spatially and semantically filtered by querying the VLM for task-relevant affordance regions on each object, as well as for arm allocation to the individual objects, thereby ensuring geometric validity while respecting task semantics. We evaluate our method on a dual-arm platform across nine real-world manipulation tasks spanning four categories: parallel manipulation, coordinated stabilization, tool use, and human handover. Our approach achieves consistently higher task success rates than geometric and semantic baselines for task-oriented grasping, demonstrating that explicit semantic reasoning over affordances and arm allocation helps enable reliable bimanual manipulation in unstructured environments.

Related papers

• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning [99.49739831030388]
  We study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning.<n>We propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface.
  arXiv  Detail & Related papers  (2026-06-11T17:59:36Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• SECOND-Grasp: Semantic Contact-guided Dexterous Grasping [60.1519218638742]
  Second-Grasp (SEmantic CONtact-guided Dexterous Grasping) is a unified framework that enables robotic hands to adjust grasping strategies based on semantic reasoning.<n>We show that our approach consistently outperforms baselines in lifting success rate on both seen and unseen categories.
  arXiv  Detail & Related papers  (2026-05-13T07:37:00Z)
• BiPreManip: Learning Affordance-Based Bimanual Preparatory Manipulation through Anticipatory Collaboration [35.18224449308742]
  Collaborative Preparatory Manipulation is a class of bimanual manipulation tasks that demand understanding object semantics and geometry.<n>We propose a visual affordance-based framework that first envisions the final goal-directed action and then guides one arm to perform a sequence of preparatory manipulations.<n>This affordance-centric representation enables anticipatory inter-arm reasoning and coordination, generalizing effectively across various objects.
  arXiv  Detail & Related papers  (2026-03-23T08:12:24Z)
• AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis [50.793806818677716]
  AffordGrasp produces physically stable and semantically faithful human grasps with high precision.<n>AffordGrasp integrates an affordance-aware latent representation of hand poses with a dual-conditioning diffusion process.<n>We evaluate AffordGrasp across four instruction-augmented benchmarks derived from HO-3D, OakInk, GRAB, and AffordPose.
  arXiv  Detail & Related papers  (2026-03-09T06:56:35Z)
• RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation [35.68205801897266]
  RoboPCA is a pose-centered affordance prediction framework that jointly predicts task-appropriate contact regions and poses conditioned on instructions.<n>Human2Afford is a data curation pipeline that automatically recovers scene-level 3D information and infers pose-centered affordance annotations from human demonstrations.<n>RoboPCA outperforms baseline methods on image datasets, simulation, and real robots, and exhibits strong generalization across tasks and categories.
  arXiv  Detail & Related papers  (2026-03-08T15:46:04Z)
• Scene-agnostic Hierarchical Bimanual Task Planning via Visual Affordance Reasoning [5.296623863894165]
  Embodied agents operating in open environments must translate high-level instructions into grounded, executable behaviors.<n>We present a unified framework for scene-agnostic bimanual task planning that bridges high-level reasoning with 3D-grounded two-handed execution.
  arXiv  Detail & Related papers  (2025-12-10T04:37:07Z)
• SpatialActor: Exploring Disentangled Spatial Representations for Robust Robotic Manipulation [63.48859753472547]
  SpatialActor is a framework for robust robotic manipulation that explicitly decouples semantics and geometry.<n>It achieves state-of-the-art performance with 87.4% on RLBench and improves by 13.9% to 19.4% under varying noisy conditions.
  arXiv  Detail & Related papers  (2025-11-12T18:59:08Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulation [14.311585896189506]
  We propose Primitive-Aware Semantic Grounding (PASG) to bridge the gap between task semantics and geometric features.<n>We demonstrate PASG's effectiveness in practical robotic manipulation tasks across diverse scenarios, achieving performance comparable to manual annotations.
  arXiv  Detail & Related papers  (2025-08-08T03:23:33Z)
• Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control [22.74768543283102]
  Graph-Fused Vision-Language-Action (GF-VLA) is a framework that enables dual-arm robotic systems to perform task-level reasoning and execution.<n>GF-VLA first extracts Shannon-information-based cues to identify hands and objects with the highest task relevance.<n>Cross-hand selection policy infers optimal assignment without explicit geometric reasoning.
  arXiv  Detail & Related papers  (2025-08-07T12:48:09Z)
• Object Affordance Recognition and Grounding via Multi-scale Cross-modal Representation Learning [64.32618490065117]
  A core problem of Embodied AI is to learn object manipulation from observation, as humans do.<n>We propose a novel approach that learns an affordance-aware 3D representation and employs a stage-wise inference strategy.<n> Experiments demonstrate the effectiveness of our method, showing improved performance in both affordance grounding and classification.
  arXiv  Detail & Related papers  (2025-08-02T04:14:18Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• ManiPose: A Comprehensive Benchmark for Pose-aware Object Manipulation in Robotics [55.85916671269219]
  This paper introduces ManiPose, a pioneering benchmark designed to advance the study of pose-varying manipulation tasks. A comprehensive dataset features geometrically consistent and manipulation-oriented 6D pose labels for 2936 real-world scanned rigid objects and 100 articulated objects. Our benchmark demonstrates notable advancements in pose estimation, pose-aware manipulation, and real-robot skill transfer.
  arXiv  Detail & Related papers  (2024-03-20T07:48:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.