Data Pyramid for Embodied Manipulation
Abstract Overview
This paper is a data-centric survey of embodied manipulation that argues embodied agents need supervision coupling observations with physical states and actions, unlike internet-scale vision-language models. The authors organize the embodied data ecosystem into a five-layer "pyramid" consisting of real-robot data, UMI-style data, egocentric and exocentric human data, simulation data, and general vision-language data. They frame these sources mainly through the trade-off between scalability and robot alignment, and further compare them using quality, diversity, reusability, and physical fidelity. The paper also reviews how embodied foundation models combine these heterogeneous sources for perception, reasoning, planning, action generation, and world prediction, and closes with open challenges such as tactile data, failure-recovery data, scalable collection, and cross-embodiment alignment.
Novelty
The paper's main novelty is a category-level "data pyramid" taxonomy for embodied manipulation that systematically positions five complementary data sources according to scalability and robot alignment. It also contributes a data-centric reading of recent embodied foundation models by relating their training recipes and alignment strategies to the capabilities they support.
Results
Rather than introducing a new manipulation algorithm, the paper delivers a structured synthesis of the embodied data landscape, including dataset categories, collection pipelines, supervision types, and transfer issues across embodiments. It shows a trend from early systems relying mostly on real-robot trajectories toward more heterogeneous mixtures of robot, UMI, egocentric, simulation, and general data, and distills key open problems for future data design.
Key Points
- The paper organizes embodied manipulation data into five complementary layers: real-robot, UMI-style, egocentric/exocentric, simulation, and general data.
- Its central analytical lens is the trade-off between scalability and robot alignment, complemented by quality, diversity, reusability, and physical fidelity.
- It analyzes how different embodied foundation model families use mixed data recipes and identifies open challenges including tactile datasets, failure-recovery data, scalable collection, and cross-embodiment alignment.