Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation

• URL: http://arxiv.org/abs/2606.04172v1
• Date: Tue, 02 Jun 2026 19:36:13 GMT
• Title: Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation
• Abstract summary: Affordance2Action (A2A) is a benchmark-centered learning framework for scene-level, task-conditioned part affordance grounding.<n>A2A-Bench is a manipulation-oriented benchmark that covers both single-region and multi-region instruction correspondences in everyday scenes.<n>A2A exposes substantial gaps in generic segmentation, VLM-based grounding, and affordance distillation baselines.
• Score: 18.782084217347187
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Task-conditioned manipulation requires grounding instructions to task-relevant functional parts rather than object categories. This setting is scene-dependent and often one-to-many in cluttered scenes: the same object may afford different interactions across tasks, while a single task may correspond to either one functional region or multiple valid functional regions, depending on the scene layout. Existing affordance datasets and benchmarks remain misaligned with this setting, as they typically focus on grasping or object-level affordances, rely on synthetic scenes, or assume a single instruction-region correspondence. We present Affordance2Action (A2A), a benchmark-centered learning framework for scene-level, task-conditioned part affordance grounding. At its core is A2A-Bench, a manipulation-oriented benchmark that covers both single-region and multi-region instruction correspondences in everyday scenes, with the latter highlighting the ambiguity and diversity of affordance grounding in realistic multi-object environments. To construct it at scale, we build A2A-AffordGen, an agent-assisted annotation pipeline that combines language-model filtering, interactive part segmentation, instance-level mask-out refinement, task-reasoning instruction generation, and human verification. A2A-Bench's supervision further supports diverse downstream applications, with real-time affordance grounding and affordance-conditioned manipulation policies as two representative examples. Experiments show that A2A exposes substantial gaps in generic segmentation, VLM-based grounding, and affordance distillation baselines, while improving task-level localization and providing useful spatial priors for downstream manipulation. All datasets and code will be publicly released to promote open research.

Related papers

• Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning [22.1291953459429]
  Bimanual manipulation requires reasoning about where to interact with an object and which arm should perform each action.<n>Existing approaches either treat affordance prediction as coarse part segmentation or rely on geometrics for arm assignment.<n>We propose a hierarchical framework for task-aware bimanual affordance prediction.
  arXiv  Detail & Related papers  (2026-04-09T19:39:45Z)
• Scene-agnostic Hierarchical Bimanual Task Planning via Visual Affordance Reasoning [5.296623863894165]
  Embodied agents operating in open environments must translate high-level instructions into grounded, executable behaviors.<n>We present a unified framework for scene-agnostic bimanual task planning that bridges high-level reasoning with 3D-grounded two-handed execution.
  arXiv  Detail & Related papers  (2025-12-10T04:37:07Z)
• ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videos [44.050522958181496]
  ToG-Bench is the first task-oriented intrinsic-temporal video grounding benchmark for ego videos.<n>Built upon videos sourced from ScanNet, ToG-Bench comprises 100 annotated clips with 2,704 task-oriented grounding instructions.<n>Extensive experiments reveal the challenges of task-oriented STVG and substantial performance gaps across explicit-implicit and multi-object grounding.
  arXiv  Detail & Related papers  (2025-12-03T10:54:44Z)
• 2HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videos [6.562256987706128]
  We propose a framework for extracting affordance data from human activity video datasets.<n>We present a VLM-based affordance prediction model, 2HandedAfforder, trained on the dataset.
  arXiv  Detail & Related papers  (2025-03-12T12:12:07Z)
• Pilot: Building the Federated Multimodal Instruction Tuning Framework [79.56362403673354]
  Our framework integrates two stages of "adapter on adapter" into the connector of the vision encoder and the LLM.<n>In stage 1, we extract task-specific features and client-specific features from visual information.<n>In stage 2, we build the cross-task Mixture-of-Adapters(CT-MoA) module to perform cross-task interaction.
  arXiv  Detail & Related papers  (2025-01-23T07:49:24Z)
• Multi-Modal Domain Adaptation Across Video Scenes for Temporal Video Grounding [59.599378814835205]
  Temporal Video Grounding (TVG) aims to localize the temporal boundary of a specific segment in an untrimmed video based on a given language query. We introduce a novel AMDA method to adaptively adjust the model's scene-related knowledge by incorporating insights from the target data.
  arXiv  Detail & Related papers  (2023-12-21T07:49:27Z)
• Exploring Relational Context for Multi-Task Dense Prediction [76.86090370115]
  We consider a multi-task environment for dense prediction tasks, represented by a common backbone and independent task-specific heads. We explore various attention-based contexts, such as global and local, in the multi-task setting. We propose an Adaptive Task-Relational Context module, which samples the pool of all available contexts for each task pair.
  arXiv  Detail & Related papers  (2021-04-28T16:45:56Z)
• Weakly Supervised Temporal Action Localization Through Learning Explicit Subspaces for Action and Context [151.23835595907596]
  Methods learn to localize temporal starts and ends of action instances in a video under only video-level supervision. We introduce a framework that learns two feature subspaces respectively for actions and their context. The proposed approach outperforms state-of-the-art WS-TAL methods on three benchmarks.
  arXiv  Detail & Related papers  (2021-03-30T08:26:53Z)
• ACSNet: Action-Context Separation Network for Weakly Supervised Temporal Action Localization [148.55210919689986]
  We introduce an Action-Context Separation Network (ACSNet) that takes into account context for accurate action localization. ACSNet outperforms existing state-of-the-art WS-TAL methods by a large margin.
  arXiv  Detail & Related papers  (2021-03-28T09:20:54Z)
• iFAN: Image-Instance Full Alignment Networks for Adaptive Object Detection [48.83883375118966]
  iFAN aims to precisely align feature distributions on both image and instance levels. It outperforms state-of-the-art methods with a boost of 10%+ AP over the source-only baseline.
  arXiv  Detail & Related papers  (2020-03-09T13:27:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.