HeteroGenManip: Generalizable Manipulation For Heterogeneous Object Interactions

• URL: http://arxiv.org/abs/2605.10201v2
• Date: Tue, 12 May 2026 07:49:53 GMT
• Title: HeteroGenManip: Generalizable Manipulation For Heterogeneous Object Interactions
• Abstract summary: Generalizable manipulation involving cross-type object interactions is a critical yet challenging capability in robotics.<n>Existing foundation-model-based approaches often adopt end-to-end learning that obscures the distinction between these stages.<n>We propose HeteroGenManip, a task-conditioned, two-stage framework designed to decouple initial grasp from complex interaction execution.
• Score: 23.788389187142446
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generalizable manipulation involving cross-type object interactions is a critical yet challenging capability in robotics. To reliably accomplish such tasks, robots must address two fundamental challenges: "where to manipulate" (contact point localization) and "how to manipulate" (subsequent interaction trajectory planning). Existing foundation-model-based approaches often adopt end-to-end learning that obscures the distinction between these stages, exacerbating error accumulation in long-horizon tasks. Furthermore, they typically rely on a single uniform model, which fails to capture the diverse, category-specific features required for heterogeneous objects. To overcome these limitations, we propose HeteroGenManip, a task-conditioned, two-stage framework designed to decouple initial grasp from complex interaction execution. First, Foundation-Correspondence-Guided Grasp module leverages structural priors to align the initial contact state, thereby significantly reducing the pose uncertainty of grasping. Subsequently, Multi-Foundation-Model Diffusion Policy (MFMDP) routes objects to category-specialized foundation models, integrating fine-grained geometric information with highly-variable part features via a dual-stream cross-attention mechanism. Experimental evaluations demonstrate that HeteroGenManip achieves robust intra-category shape and pose generalization. The framework achieves an average 31% performance improvement in simulation tasks with broad type setting, alongside a 36.7% gain across four real-world tasks with different interaction types.

Related papers

• ArtManip: Category-Level Articulated In-Hand Manipulation [38.430290236289004]
  Category-level in-hand manipulation of articulated objects is a formidable yet underexplored challenge for robotic hands.<n>We present ArtManip, the first category-level articulated in-hand manipulation method that generalizes across object instances and diverse initial grasps.
  arXiv  Detail & Related papers  (2026-09-11T06:59:21Z)
• Structure over Depth: A Single-Block Spatio-Temporal Transformer for Multi-Entity Reasoning [0.0]
  Multi-entity temporal data requires capturing dependencies across entities, time, and their interactions.<n> Transformer-based approaches decompose well but often rely on deep stacks of layers to learn these heterogeneous dependencies implicitly.<n>We propose a structured-temporal-self-attention transformer block that explicitly models all three within a single stage.<n>We evaluate the approach on video-based group activity recognition, skeleton-based human interaction analysis, and wearable sensor-based activity recognition.
  arXiv  Detail & Related papers  (2026-07-25T07:02:07Z)
• Structural Action Transformer for 3D Dexterous Manipulation [80.07649565189035]
  Cross-embodiment skill transfer is a challenge for high-DoF robotic hands.<n>Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations.<n>This paper proposes a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective.
  arXiv  Detail & Related papers  (2026-03-04T11:38:12Z)
• Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing [20.40288070674112]
  We propose an end-to-end Interaction-aware Transformer (InterFormer)<n>It integrates three key components, i.e., a Dynamic Query Generator (DQG), a Dual-context Feature Selector (DFS), and the Conditional Co-occurrence (CoCo) loss.<n>Our model achieves state-of-the-art performance on both the EgoHOS and the challenging out-of-distribution mini-HOI4D datasets.
  arXiv  Detail & Related papers  (2026-02-24T06:39:18Z)
• HeLoFusion: An Efficient and Scalable Encoder for Modeling Heterogeneous and Multi-Scale Interactions in Trajectory Prediction [11.30785902722196]
  HeLoFusion is an efficient and scalable encoder for modeling heterogeneous and multi-scale agent interactions.<n>Our work demonstrates that a locality-grounded architecture, which explicitly models multi-scale and heterogeneous interactions, is a highly effective strategy for advancing motion forecasting.
  arXiv  Detail & Related papers  (2025-09-15T09:19:41Z)
• Human-in-Context: Unified Cross-Domain 3D Human Motion Modeling via In-Context Learning [64.30639042094548]
  We propose a new setting to train a unified cross-domain model through a single process.<n>We first introduce Pose-in-Context (PiC), which leverages in-context learning to create a pose-centric cross-domain model.<n>We thus propose Human-in-Context (HiC), an extension of PiC that broadens generalization across modalities, tasks, and datasets.
  arXiv  Detail & Related papers  (2025-08-14T17:59:23Z)
• Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Grounding [18.52792284421002]
  Articulated objects pose diverse manipulation challenges for robots.<n>Since their internal structures are not directly observable, robots must adaptively explore and refine actions to generate successful manipulation trajectories.<n>AdaRPG is a novel framework that leverages foundation models to extract object parts, which exhibit greater local geometric similarity than entire objects.
  arXiv  Detail & Related papers  (2025-07-24T10:25:58Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection [116.21529970404653]
  We introduce SG2HOI+, a unified one-step model based on the Transformer architecture. Our approach employs two interactive hierarchical Transformers to seamlessly unify the tasks of SGG and HOI detection. Our approach achieves competitive performance when compared to state-of-the-art HOI methods.
  arXiv  Detail & Related papers  (2023-11-03T07:25:57Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models.<n>Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning.<n>Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)
• Modeling Instance Interactions for Joint Information Extraction with Neural High-Order Conditional Random Field [39.055053720433435]
  We introduce a joint IE framework (CRFIE) that formulates joint IE as a high-order Conditional Random Field. Specifically, we design binary factors and ternary factors to directly model interactions between not only a pair of instances but also triplets. We incorporate a high-order neural decoder that is unfolded from a mean-field variational inference method.
  arXiv  Detail & Related papers  (2022-12-17T18:45:23Z)
• Decoupled Multi-task Learning with Cyclical Self-Regulation for Face Parsing [71.19528222206088]
  We propose a novel Decoupled Multi-task Learning with Cyclical Self-Regulation for face parsing. Specifically, DML-CSR designs a multi-task model which comprises face parsing, binary edge, and category edge detection. Our method achieves the new state-of-the-art performance on the Helen, CelebA-HQ, and LapaMask datasets.
  arXiv  Detail & Related papers  (2022-03-28T02:12:30Z)
• Cascaded Human-Object Interaction Recognition [175.60439054047043]
  We introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network. With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding.
  arXiv  Detail & Related papers  (2020-03-09T17:05:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.