TriRelVLA: Triadic Relational Structure for Generalizable Embodied Manipulation

• URL: http://arxiv.org/abs/2605.05714v1
• Date: Thu, 07 May 2026 05:57:49 GMT
• Title: TriRelVLA: Triadic Relational Structure for Generalizable Embodied Manipulation
• Abstract summary: Vision-language--action (VLA) models perform well on training-seen robotic tasks but struggle to generalize to unseen scenes and objects.<n>We propose TriRelVLA, a triadic relational VLA framework for generalizable embodied manipulation.<n> Experiments show strong performance on fine-tuned tasks and clear gains in cross-scene, cross-object, and cross-task generalization.
• Score: 54.81449795163812
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language-action (VLA) models perform well on training-seen robotic tasks but struggle to generalize to unseen scenes and objects. A key limitation lies in their implicit visual representations, which entangle object appearance, background, and scene layout. This makes policies sensitive to visual variations. Prior work improves transferability through structured intermediate representations that objectify visual content. However, these representations mainly capture scene semantics instead of action-relevant relations. As a result, action prediction remains tied to appearance statistics. We observe that manipulation actions depend on the object-hand-task relational structure, which governs interactions among task requirements, robot states, and object properties. Based on this observation, we propose TriRelVLA, a triadic relational VLA framework for generalizable embodied manipulation. Our approach consists of three components: 1) We construct explicit object-hand-task triadic representations from multimodal inputs as relational primitives. 2) We build a task-grounded relational graph. Task-guided cross-attention forms nodes, and a relation-aware graph transformer models interactions among them. 3) We perform relation-conditioned action generation. The relational structure is compressed into a bottleneck space and projected into the LLM for action prediction. This triadic relational bottleneck reduces reliance on appearance statistics and enables transfer across scenes, objects, and task compositions. We further introduce a real-world robotic dataset for fine-tuning. Experiments show strong performance on fine-tuned tasks and clear gains in cross-scene, cross-object, and cross-task generalization.

Related papers

• FOCI Policy: Focus on Object-Centric Interactions for Relational Manipulation Policies [6.557127548028966]
  Object-centric manipulation policies improve generalization by modeling object motion instead of directly predicting robot actions.<n>We propose textscFoci Policy, an interaction-centric framework that achieves a two-fold abstraction.<n>Experiments on RLBench, COLOSSEUM, and real-world tasks show that textscFoci Policy achieves strong performance with substantially less training data than prior object-centric and action-centric policies.
  arXiv  Detail & Related papers  (2026-09-08T13:35:56Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning [27.511627003202538]
  Traditional scene graphs primarily focus on spatial relationships, limiting vision-language models' (VLMs) ability to reason about complex interactions in visual scenes.<n>This paper addresses two key challenges: (1) conventional detection-to-construction methods produce unfocused, contextually irrelevant relationship sets, and (2) existing approaches fail to form persistent memories for generalizing interaction reasoning to new scenes.<n>We propose Interaction-augmented Scene Graph Reasoning (ISGR), a framework that enhances VLMs' interactional reasoning through three complementary components.
  arXiv  Detail & Related papers  (2025-05-14T04:04:23Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Situational Scene Graph for Structured Human-centric Situation Understanding [15.91717913059569]
  We propose a graph-based representation called Situational Scene Graph (SSG) to encode both humanobject-relationships and the corresponding semantic properties.<n>The semantic details are represented as predefined roles and values inspired by situation frame, which is originally designed to represent a single action.<n>We will release the code and the dataset soon.
  arXiv  Detail & Related papers  (2024-10-30T09:11:25Z)
• OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion [39.14950571922401]
  OAKINK2 is a dataset of bimanual object manipulation tasks for complex daily activities. It introduces three level of abstraction to organize the manipulation tasks: Affordance, Primitive Task, and Complex Task. OakINK2 dataset provides multi-view image streams and precise pose annotations for the human body, hands and various interacting objects.
  arXiv  Detail & Related papers  (2024-03-28T13:47:19Z)
• Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection [116.21529970404653]
  We introduce SG2HOI+, a unified one-step model based on the Transformer architecture. Our approach employs two interactive hierarchical Transformers to seamlessly unify the tasks of SGG and HOI detection. Our approach achieves competitive performance when compared to state-of-the-art HOI methods.
  arXiv  Detail & Related papers  (2023-11-03T07:25:57Z)
• Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge [62.981429762309226]
  The ability to actively ground task instructions from an egocentric view is crucial for AI agents to accomplish tasks or assist humans virtually. We propose to improve phrase grounding models' ability on localizing the active objects by: learning the role of objects undergoing change and extracting them accurately from the instructions. We evaluate our framework on Ego4D and Epic-Kitchens datasets.
  arXiv  Detail & Related papers  (2023-10-23T16:14:05Z)
• Learning Action-Effect Dynamics from Pairs of Scene-graphs [50.72283841720014]
  We propose a novel method that leverages scene-graph representation of images to reason about the effects of actions described in natural language. Our proposed approach is effective in terms of performance, data efficiency, and generalization capability compared to existing models.
  arXiv  Detail & Related papers  (2022-12-07T03:36:37Z)
• RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning [139.0548263507796]
  We use vision transformers (ViTs) as our base model for visual reasoning. We make better use of concepts defined as object entities and their relations to improve the reasoning ability of ViTs. We show the resulting model, Concept-guided Vision Transformer (or RelViT for short), significantly outperforms prior approaches on HICO and GQA benchmarks.
  arXiv  Detail & Related papers  (2022-04-24T02:46:43Z)
• SORNet: Spatial Object-Centric Representations for Sequential Manipulation [39.88239245446054]
  Sequential manipulation tasks require a robot to perceive the state of an environment and plan a sequence of actions leading to a desired goal state. We propose SORNet, which extracts object-centric representations from RGB images conditioned on canonical views of the objects of interest.
  arXiv  Detail & Related papers  (2021-09-08T19:36:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.