Compositional Motion Generation from Demonstration with Object-Centric Neural Fields

• URL: http://arxiv.org/abs/2607.07129v1
• Date: Wed, 08 Jul 2026 08:20:24 GMT
• Title: Compositional Motion Generation from Demonstration with Object-Centric Neural Fields
• Abstract summary: We propose a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior.<n>We render scenes from object-centric neural representations that integrate canonical neural fields with latent-conditioned deformations.<n>In simulation, long-horizon manipulation tasks are successfully completed using the proposed model.
• Score: 1.4916971861796382
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compositionality, by organizing complex behavior as combinations of simpler elements, enables robot learning that is scalable and data efficient. Leveraging this principle, we propose a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior by connecting perception and motion through shared object-level representations. We render scenes from object-centric neural representations that integrate canonical neural fields with latent-conditioned deformations, capturing positional and geometric variations in a smooth, consistent, and interpretable way. For motion generation, a temporal mixture-of-experts (MoE) employs a gating mechanism to combine object-conditioned movement primitives over time, producing complete trajectories. This spatial-temporal compositionality maintains the data efficiency of movement primitives while grounding motion in visual structure, enabling systematic generalization across diverse scene configurations. In simulation, long-horizon manipulation tasks are successfully completed using the proposed model, which requires significantly less training data than other image-based baselines. Real-world experiments further demonstrate the method's robustness to noise, its ability to generalize at the category level through language-based segmentation models, and its capacity to operate directly on 3D scene representations.

Related papers

• RORA: Realistic Object Reconstruction with Articulation [10.305820103142269]
  We propose the first end-to-end pipeline that reconstructs simulation-ready assets with accurate articulation from a single static object video input.<n>Our approach exports a hybrid representation combining 3DGS for rendering and mesh-based geometry for physical interaction.<n>We have shown that our method achieves precise articulation results on partnet-mobility-v0 dataset and real objects.
  arXiv  Detail & Related papers  (2026-08-05T13:36:58Z)
• GenMatter: Perceiving Physical Objects with Generative Matter Models [50.121713214509604]
  Humans robustly segment moving entities that constitute independently moveable chunks of matter.<n>Existing computer vision systems lack a unified approach that works across diverse settings.<n>We propose a generative model that hierarchically groups low-level motion cues and high-level appearance features into particles.
  arXiv  Detail & Related papers  (2026-04-24T02:18:18Z)
• ArtLLM: Generating Articulated Assets via 3D LLM [19.814132638278547]
  ArtLLM is a novel framework for generating high-quality articulated assets directly from complete 3D meshes.<n>At its core is a 3D multimodal large language model trained on a large-scale articulation dataset.<n> Experiments show that ArtLLM significantly outperforms state-of-the-art methods in both part layout accuracy and joint prediction.
  arXiv  Detail & Related papers  (2026-03-01T15:07:46Z)
• Kinematify: Open-Vocabulary Synthesis of High-DoF Articulated Objects [59.51185639557874]
  We introduce Kinematify, an automated framework that synthesizes articulated objects directly from arbitrary RGB images or textual descriptions.<n>Our method addresses two core challenges: (i) inferring kinematic topologies for high-DoF objects and (ii) estimating joint parameters from static geometry.
  arXiv  Detail & Related papers  (2025-11-03T07:21:42Z)
• HEIR: Learning Graph-Based Motion Hierarchies [30.232977288956814]
  We propose a general hierarchical motion modeling method that learns structured, interpretable motion relationships directly from data.<n>Our method represents observed motions using graph-based hierarchies, explicitly decomposing global absolute motions into parent-inherited patterns and local motion residuals.<n> Experimental results show that our method reconstructs the intrinsic motion hierarchy in 1D and 2D cases, and produces more realistic and interpretable deformations compared to the baseline on dynamic 3D splatting scenes.
  arXiv  Detail & Related papers  (2025-10-30T17:57:40Z)
• GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects [4.717906057951389]
  We introduce a unified representation that jointly models geometry and motion using articulated 3D Gaussians.<n>This formulation improves robustness in motion decomposition and supports articulated objects with up to 20 parts.<n>We show that our method consistently achieves superior accuracy in part-level geometry reconstruction and motion estimation across a broad range of object types.
  arXiv  Detail & Related papers  (2025-08-20T17:59:08Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in video [3.2195139886901813]
  We present a novel framework that learns multi-level semantical dynamics and multi-frame human pose estimation.<n>Specifically, we first design a multi-masked context and pose reconstruction strategy.<n>This strategy stimulates the model to explore multi-temporal semantic relationships among frames by progressively masking the features of optical (patch) cubes and frames.
  arXiv  Detail & Related papers  (2025-02-15T00:35:34Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• Relational Object-Centric Actor-Critic [44.99833362998488]
  Recent works highlight that disentangled object representations can aid policy learning in image-based, object-centric reinforcement learning tasks.<n>This paper proposes a novel object-centric reinforcement learning algorithm that integrates actor-critic and model-based approaches.<n>We evaluate our method in a simulated 3D robotic environment and a 2D environment with compositional structure.
  arXiv  Detail & Related papers  (2023-10-26T06:05:12Z)
• Robust and Controllable Object-Centric Learning through Energy-based Models [95.68748828339059]
  ours is a conceptually simple and general approach to learning object-centric representations through an energy-based model. We show that ours can be easily integrated into existing architectures and can effectively extract high-quality object-centric representations.
  arXiv  Detail & Related papers  (2022-10-11T15:11:15Z)
• MoDi: Unconditional Motion Synthesis from Diverse Data [51.676055380546494]
  We present MoDi, an unconditional generative model that synthesizes diverse motions. Our model is trained in a completely unsupervised setting from a diverse, unstructured and unlabeled motion dataset. We show that despite the lack of any structure in the dataset, the latent space can be semantically clustered.
  arXiv  Detail & Related papers  (2022-06-16T09:06:25Z)
• Conditional Object-Centric Learning from Video [34.012087337046005]
  We introduce a sequential extension to Slot Attention to predict optical flow for realistic looking synthetic scenes. We show that conditioning the initial state of this model on a small set of hints, such as center of mass of objects in the first frame, is sufficient to significantly improve instance segmentation. These benefits generalize beyond the training distribution to novel objects, novel backgrounds, and to longer video sequences.
  arXiv  Detail & Related papers  (2021-11-24T16:10:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.