Scene-agnostic Hierarchical Bimanual Task Planning via Visual Affordance Reasoning

• URL: http://arxiv.org/abs/2512.09310v1
• Date: Wed, 10 Dec 2025 04:37:07 GMT
• Title: Scene-agnostic Hierarchical Bimanual Task Planning via Visual Affordance Reasoning
• Abstract summary: Embodied agents operating in open environments must translate high-level instructions into grounded, executable behaviors.<n>We present a unified framework for scene-agnostic bimanual task planning that bridges high-level reasoning with 3D-grounded two-handed execution.
• Score: 5.296623863894165
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied agents operating in open environments must translate high-level instructions into grounded, executable behaviors, often requiring coordinated use of both hands. While recent foundation models offer strong semantic reasoning, existing robotic task planners remain predominantly unimanual and fail to address the spatial, geometric, and coordination challenges inherent to bimanual manipulation in scene-agnostic settings. We present a unified framework for scene-agnostic bimanual task planning that bridges high-level reasoning with 3D-grounded two-handed execution. Our approach integrates three key modules. Visual Point Grounding (VPG) analyzes a single scene image to detect relevant objects and generate world-aligned interaction points. Bimanual Subgoal Planner (BSP) reasons over spatial adjacency and cross-object accessibility to produce compact, motion-neutralized subgoals that exploit opportunities for coordinated two-handed actions. Interaction-Point-Driven Bimanual Prompting (IPBP) binds these subgoals to a structured skill library, instantiating synchronized unimanual or bimanual action sequences that satisfy hand-state and affordance constraints. Together, these modules enable agents to plan semantically meaningful, physically feasible, and parallelizable two-handed behaviors in cluttered, previously unseen scenes. Experiments show that it produces coherent, feasible, and compact two-handed plans, and generalizes to cluttered scenes without retraining, demonstrating robust scene-agnostic affordance reasoning for bimanual tasks.

Related papers

• Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation [0.6927055673104934]
  The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution.<n>The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning.
  arXiv  Detail & Related papers  (2026-06-22T11:01:56Z)
• See Selectively, Act Adaptively: Dual-Level Structural Decomposition for Bimanual Robot Manipulation [1.7587442088965224]
  We propose a bimanual manipulation framework based on Dual-Level Structural Decomposition.<n>We evaluate the proposed method on six simulated bimanual manipulation tasks in RoboTwin 2.0 and three long-horizon real-world tasks.
  arXiv  Detail & Related papers  (2026-06-11T12:33:55Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation [18.782084217347187]
  Affordance2Action (A2A) is a benchmark-centered learning framework for scene-level, task-conditioned part affordance grounding.<n>A2A-Bench is a manipulation-oriented benchmark that covers both single-region and multi-region instruction correspondences in everyday scenes.<n>A2A exposes substantial gaps in generic segmentation, VLM-based grounding, and affordance distillation baselines.
  arXiv  Detail & Related papers  (2026-06-02T19:36:13Z)
• Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spaces [116.57196064763924]
  We extend the benchmark coverage by introducing dense tabletop objects and explicit multi-level functional relationships.<n>This work introduces critical challenges involving small-scale, dense, and similar instances.<n>We propose an open-vocabulary pipeline based on 2D visual grounding and 3D graph optimization.
  arXiv  Detail & Related papers  (2026-05-15T09:14:50Z)
• Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning [22.1291953459429]
  Bimanual manipulation requires reasoning about where to interact with an object and which arm should perform each action.<n>Existing approaches either treat affordance prediction as coarse part segmentation or rely on geometrics for arm assignment.<n>We propose a hierarchical framework for task-aware bimanual affordance prediction.
  arXiv  Detail & Related papers  (2026-04-09T19:39:45Z)
• AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis [50.793806818677716]
  AffordGrasp produces physically stable and semantically faithful human grasps with high precision.<n>AffordGrasp integrates an affordance-aware latent representation of hand poses with a dual-conditioning diffusion process.<n>We evaluate AffordGrasp across four instruction-augmented benchmarks derived from HO-3D, OakInk, GRAB, and AffordPose.
  arXiv  Detail & Related papers  (2026-03-09T06:56:35Z)
• Action-Sketcher: From Reasoning to Action via Visual Sketches for Long-Horizon Robotic Manipulation [37.76888706583038]
  Long-horizon robotic manipulation is increasingly important for real-world deployment.<n>Existing end-to-end and hierarchical Vision-Language-Action policies often rely on text-only cues.<n>We introduce Visual Sketch, an implausible visual intermediate that renders points, boxes, arrows, and typed relations in the robot's current views.<n>We present Action-Sketcher, a VLA framework that operates in a cyclic See-Think-Sketch-Act workflow.
  arXiv  Detail & Related papers  (2026-01-04T17:53:42Z)
• ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos [44.050522958181496]
  ToG-Bench is the first task-oriented intrinsic-temporal video grounding benchmark for ego videos.<n>Built upon videos sourced from ScanNet, ToG-Bench comprises 100 annotated clips with 2,704 task-oriented grounding instructions.<n>Extensive experiments reveal the challenges of task-oriented STVG and substantial performance gaps across explicit-implicit and multi-object grounding.
  arXiv  Detail & Related papers  (2025-12-03T10:54:44Z)
• SAGE: Scene Graph-Aware Guidance and Execution for Long-Horizon Manipulation Tasks [3.688836621357062]
  Long-horizon manipulation tasks involve extended action sequences and complex object interactions.<n>We propose SAGE, a novel framework for Scene Graph-Aware Guidance and Execution in Long-Horizon Manipulation Tasks.<n> SAGE consists of two key components: (1) a scene graph-based task planner that uses VLMs and LLMs to parse the environment and reason about physically-grounded scene state transition sequences, and (2) a decoupled structural image editing pipeline that controllably converts each target sub-goal graph into a corresponding image.
  arXiv  Detail & Related papers  (2025-09-26T06:14:55Z)
• GeoSAM2: Unleashing the Power of SAM2 for 3D Part Segmentation [81.0871900167463]
  We introduce GeoSAM2, a prompt-controllable framework for 3D part segmentation.<n>Given a textureless object, we render normal and point maps from predefined viewpoints.<n>We accept simple 2D prompts - clicks or boxes - to guide part selection.<n>The predicted masks are back-projected to the object and aggregated across views.
  arXiv  Detail & Related papers  (2025-08-19T17:58:51Z)
• PASG: A Closed-Loop Framework for Automated Geometric Primitive Extraction and Semantic Anchoring in Robotic Manipulation [14.311585896189506]
  We propose Primitive-Aware Semantic Grounding (PASG) to bridge the gap between task semantics and geometric features.<n>We demonstrate PASG's effectiveness in practical robotic manipulation tasks across diverse scenarios, achieving performance comparable to manual annotations.
  arXiv  Detail & Related papers  (2025-08-08T03:23:33Z)
• HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception [57.37135310143126]
  HO SIG is a novel framework for synthesizing full-body interactions through hierarchical scene perception.<n>Our framework supports unlimited motion length through autoregressive generation and requires minimal manual intervention.<n>This work bridges the critical gap between scene-aware navigation and dexterous object manipulation.
  arXiv  Detail & Related papers  (2025-06-02T12:08:08Z)
• Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planning [103.24305074625106]
  We propose 3D Reasoning-Driven Planning, a novel 3D task that reasons the intended activities from implicit instructions and decomposes them into steps with inter-step routes and planning.<n>First, we construct ReasonPlan3D, a large-scale benchmark that covers diverse 3D scenes with rich implicit instructions.<n>Second, we design a novel framework that introduces progressive plan generation with contextual consistency across multiple steps.
  arXiv  Detail & Related papers  (2025-03-17T09:33:58Z)
• ManiPose: A Comprehensive Benchmark for Pose-aware Object Manipulation in Robotics [55.85916671269219]
  This paper introduces ManiPose, a pioneering benchmark designed to advance the study of pose-varying manipulation tasks. A comprehensive dataset features geometrically consistent and manipulation-oriented 6D pose labels for 2936 real-world scanned rigid objects and 100 articulated objects. Our benchmark demonstrates notable advancements in pose estimation, pose-aware manipulation, and real-robot skill transfer.
  arXiv  Detail & Related papers  (2024-03-20T07:48:32Z)
• Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap [72.01609575400498]
  We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces. We propose a Latent Space Roadmap (LSR) for task planning, a graph-based structure capturing globally the system dynamics in a low-dimensional latent space. We present a thorough investigation of our framework on two simulated box stacking tasks and a folding task executed on a real robot.
  arXiv  Detail & Related papers  (2021-03-03T17:48:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.