Iterative Compositional Data Generation for Robot Control

• URL: http://arxiv.org/abs/2512.10891v2
• Date: Fri, 12 Dec 2025 15:43:07 GMT
• Title: Iterative Compositional Data Generation for Robot Control
• Authors: Anh-Quan Pham, Marcel Hussing, Shubhankar P. Patankar, Dani S. Bassett, Jorge Mendez-Mendez, Eric Eaton,
• Abstract summary: We propose a semantic compositional diffusion transformer that factorizes transitions into robot-, object-, obstacle-, and objective-specific components.<n>We show that our model can zero-shot generate high-quality transitions from which we can learn control policies for unseen task combinations.
• Score: 8.888011063034517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Collecting robotic manipulation data is expensive, making it impractical to acquire demonstrations for the combinatorially large space of tasks that arise in multi-object, multi-robot, and multi-environment settings. While recent generative models can synthesize useful data for individual tasks, they do not exploit the compositional structure of robotic domains and struggle to generalize to unseen task combinations. We propose a semantic compositional diffusion transformer that factorizes transitions into robot-, object-, obstacle-, and objective-specific components and learns their interactions through attention. Once trained on a limited subset of tasks, we show that our model can zero-shot generate high-quality transitions from which we can learn control policies for unseen task combinations. Then, we introduce an iterative self-improvement procedure in which synthetic data is validated via offline reinforcement learning and incorporated into subsequent training rounds. Our approach substantially improves zero-shot performance over monolithic and hard-coded compositional baselines, ultimately solving nearly all held-out tasks and demonstrating the emergence of meaningful compositional structure in the learned representations.

Related papers

• Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards [13.784988950752195]
  Existing open-source agentic training data are narrow in task variety and easily solved.<n>Real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes.<n>We address these challenges with SYNTHAGENT, a framework that jointly synthesizes diverse tool-use training data and simulates complete environments.
  arXiv  Detail & Related papers  (2026-01-30T03:43:42Z)
• Learning Semantic-Geometric Task Graph-Representations from Human Demonstrations [16.68801520494275]
  We introduce a semantic-geometric task graph-representation that encodes object identities, inter-object relations, and their temporal geometric evolution from human demonstrations.<n>We show that semantic-geometric task graph-representations are particularly beneficial for tasks with high action and object variability.
  arXiv  Detail & Related papers  (2026-01-16T17:35:00Z)
• Learning Causal Structure Distributions for Robust Planning [53.753366558072806]
  We find that learning the functional relationships while accounting for the uncertainty about the structural information leads to more robust dynamics models.<n>This in contrast with common model-learning methods that ignore the causal structure and fail to leverage the sparsity of interactions in robotic systems.<n>We show that our model can be used to learn the dynamics of a robot, which together with a sampling-based planner can be used to perform new tasks in novel environments.
  arXiv  Detail & Related papers  (2025-08-08T22:43:17Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following [50.377287115281476]
  We show that learning to associate the representations of current and future states with a temporal loss can improve compositional generalization.<n>We evaluate our approach across diverse robotic manipulation tasks as well as in simulation, showing substantial improvements for tasks specified with either language or goal images.
  arXiv  Detail & Related papers  (2025-02-08T05:26:29Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• Efficient and Robust Training of Dense Object Nets for Multi-Object Robot Manipulation [8.321536457963655]
  We propose a framework for robust and efficient training of Dense Object Nets (DON) We focus on training with multi-object data instead of singulated objects, combined with a well-chosen augmentation scheme. We demonstrate the robustness and accuracy of our proposed framework on a real-world robotic grasping task.
  arXiv  Detail & Related papers  (2022-06-24T08:24:42Z)
• Graph-based Reinforcement Learning meets Mixed Integer Programs: An application to 3D robot assembly discovery [34.25379651790627]
  We tackle the problem of building arbitrary, predefined target structures entirely from scratch using a set of Tetris-like building blocks and a robotic manipulator. Our novel hierarchical approach aims at efficiently decomposing the overall task into three feasible levels that benefit mutually from each other.
  arXiv  Detail & Related papers  (2022-03-08T14:44:51Z)
• Object Pursuit: Building a Space of Objects via Discriminative Weight Generation [23.85039747700698]
  We propose a framework to continuously learn object-centric representations for visual learning and understanding. We leverage interactions to sample diverse variations of an object and the corresponding training signals while learning the object-centric representations. We perform an extensive study of the key features of the proposed framework and analyze the characteristics of the learned representations.
  arXiv  Detail & Related papers  (2021-12-15T08:25:30Z)
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks [73.63892022944198]
  We present a generic perception architecture named Uni-Perceiver. It processes a variety of modalities and tasks with unified modeling and shared parameters. Results show that our pre-trained model without any tuning can achieve reasonable performance even on novel tasks.
  arXiv  Detail & Related papers  (2021-12-02T18:59:50Z)
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning [138.40338621974954]
  CausalWorld is a benchmark for causal structure and transfer learning in a robotic manipulation environment. Tasks consist of constructing 3D shapes from a given set of blocks - inspired by how children learn to build complex structures.
  arXiv  Detail & Related papers  (2020-10-08T23:01:13Z)
• Scalable Multi-Task Imitation Learning with Autonomous Improvement [159.9406205002599]
  We build an imitation learning system that can continuously improve through autonomous data collection. We leverage the robot's own trials as demonstrations for tasks other than the one that the robot actually attempted. In contrast to prior imitation learning approaches, our method can autonomously collect data with sparse supervision for continuous improvement.
  arXiv  Detail & Related papers  (2020-02-25T18:56:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.