Data Pyramid for Embodied Manipulation

• URL: http://arxiv.org/abs/2607.24744v1
• Date: Mon, 27 Jul 2026 17:59:58 GMT
• Title: Data Pyramid for Embodied Manipulation
• Abstract summary: Embodied agents require data that couple observations with physical states and actions.<n>We organize the embodied data ecosystem as a "pyramid" spanning five complementary sources.
• Score: 135.16063649795234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple data sources. In this work, we organize the embodied data ecosystem as a "pyramid" spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity. We then analyze recent embodied foundation models through the lens of their data recipes, examining how different sources are selected, aligned, and mixed during pretraining. For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work paves the foundation for the design of next-generation embodied systems.

Related papers

• HumanoidMimicGen: Data Generation for Loco-Manipulation via Whole-Body Planning [52.022681285103126]
  We present HumanoidMimicGen, a method for generating humanoid legged loco-manipulation data.<n>Our method adapts contact-rich whole-body skills from a handful of source demonstrations to new states.<n>We show that whole-body visuomotor policies co-trained with data generated by HumanoidMimicGen outperform those trained only on real-world data by 20%.
  arXiv  Detail & Related papers  (2026-05-26T21:57:11Z)
• SoftMimicGen: A Data Generation System for Scalable Robot Learning in Deformable Object Manipulation [45.71119158785489]
  We introduce SoftMimicGen, an automated data generation pipeline for deformable object manipulation tasks.<n>We introduce a suite of high-fidelity simulation environments that encompasses a wide range of deformable objects.<n>We apply SoftMimicGen to generate datasets across the task suite, train high-performing policies from the data, and systematically analyze the data generation system.
  arXiv  Detail & Related papers  (2026-03-26T17:58:40Z)
• What Matters in Learning from Large-Scale Datasets for Robot Manipulation [12.703188997313223]
  We conduct a large-scale dataset composition study to answer this question.<n>We develop a data generation framework to procedurally emulate common sources of diversity in existing datasets.<n>We find that camera poses and spatial arrangements are crucial dimensions for both diversity in collection and alignment in retrieval.
  arXiv  Detail & Related papers  (2025-06-16T14:25:29Z)
• Diffusion-Driven Inertial Generated Data for Smartphone Location Classification [6.24709220163167]
  We propose diffusion-driven specific force-generated data for smartphone location recognition.<n>Our results demonstrate that our diffusion-based generative model successfully captures the distinctive characteristics of specific force signals.
  arXiv  Detail & Related papers  (2025-04-20T10:14:36Z)
• DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Data [61.62554324594797]
  We propose DreamMask, which explores how to generate training data in the open-vocabulary setting, and how to train the model with both real and synthetic data.<n>In general, DreamMask significantly simplifies the collection of large-scale training data, serving as a plug-and-play enhancement for existing methods.<n>For instance, when trained on COCO and tested on ADE20K, the model equipped with DreamMask outperforms the previous state-of-the-art by a substantial margin of 2.1% mIoU.
  arXiv  Detail & Related papers  (2025-01-03T19:00:00Z)
• Can-Do! A Dataset and Neuro-Symbolic Grounded Framework for Embodied Planning with Large Multimodal Models [85.55649666025926]
  We introduce Can-Do, a benchmark dataset designed to evaluate embodied planning abilities. Our dataset includes 400 multimodal samples, each consisting of natural language user instructions, visual images depicting the environment, state changes, and corresponding action plans. We propose NeuroGround, a neurosymbolic framework that first grounds the plan generation in the perceived environment states and then leverages symbolic planning engines to augment the model-generated plans.
  arXiv  Detail & Related papers  (2024-09-22T00:30:11Z)
• Data Augmentation in Human-Centric Vision [54.97327269866757]
  This survey presents a comprehensive analysis of data augmentation techniques in human-centric vision tasks. It delves into a wide range of research areas including person ReID, human parsing, human pose estimation, and pedestrian detection. Our work categorizes data augmentation methods into two main types: data generation and data perturbation.
  arXiv  Detail & Related papers  (2024-03-13T16:05:18Z)
• Application of Federated Learning in Building a Robust COVID-19 Chest X-ray Classification Model [0.0]
  Federated Learning (FL) helps AI models to generalize better without moving all the data to a central server. We trained a deep learning model to solve a binary classification problem of predicting the presence or absence of COVID-19.
  arXiv  Detail & Related papers  (2022-04-22T05:21:50Z)
• MetaGraspNet: A Large-Scale Benchmark Dataset for Vision-driven Robotic Grasping via Physics-based Metaverse Synthesis [78.26022688167133]
  We present a large-scale benchmark dataset for vision-driven robotic grasping via physics-based metaverse synthesis. The proposed dataset contains 100,000 images and 25 different object types. We also propose a new layout-weighted performance metric alongside the dataset for evaluating object detection and segmentation performance.
  arXiv  Detail & Related papers  (2021-12-29T17:23:24Z)
• A Note on Data Biases in Generative Models [16.86600007830682]
  We investigate the impact of dataset quality on the performance of generative models. We show how societal biases of datasets are replicated by generative models. We present creative applications through unpaired transfer between diverse datasets such as photographs, oil portraits, and animes.
  arXiv  Detail & Related papers  (2020-12-04T10:46:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.