3D Generation for Embodied AI and Robotic Simulation: A Survey

• URL: http://arxiv.org/abs/2604.26509v2
• Date: Wed, 06 May 2026 14:11:41 GMT
• Title: 3D Generation for Embodied AI and Robotic Simulation: A Survey
• Abstract summary: Embodied AI and robotic systems increasingly depend on scalable, diverse, and physically grounded 3D content for simulation-based training and real-world deployment.<n>This survey reviews 3D generation for embodied AI and organizes the literature around three roles that 3D generation plays in embodied systems.<n>In Data Generator, 3D generation produces simulation-ready objects and assets, including articulated, physically grounded, and deformable content for downstream interaction.<n>In Simulation Environments, it constructs interactive and task-oriented worlds, spanning structure-aware, controllable, and agentic scene generation.<n>In Sim2Real Bridge, it supports digital twin reconstruction
• Score: 19.441445845534158
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied AI and robotic systems increasingly depend on scalable, diverse, and physically grounded 3D content for simulation-based training and real-world deployment. While 3D generative modeling has advanced rapidly, embodied applications impose requirements far beyond visual realism: generated objects must carry kinematic structure and material properties, scenes must support interaction and task execution, and the resulting content must bridge the gap between simulation and reality. This survey reviews 3D generation for embodied AI and organizes the literature around three roles that 3D generation plays in embodied systems. In Data Generator, 3D generation produces simulation-ready objects and assets, including articulated, physically grounded, and deformable content for downstream interaction; in Simulation Environments, it constructs interactive and task-oriented worlds, spanning structure-aware, controllable, and agentic scene generation; and in Sim2Real Bridge, it supports digital twin reconstruction, data augmentation, and synthetic demonstrations for downstream robot learning and real-world transfer. We also show that the field is shifting from visual realism toward interaction readiness, and we identify the main bottlenecks, including limited physical annotations, the gap between geometric quality and physical validity, fragmented evaluation, and the persistent sim-to-real divide, that must be addressed for 3D generation to become a dependable foundation for embodied intelligence. Our project page is at https://3dgen4robot.github.io.

Related papers

• V-Dreamer: Automating Robotic Simulation and Trajectory Synthesis via Video Generation Priors [30.872127145725187]
  V-Dreamer is a fully automated framework that generates open-vocabulary, simulation-ready manipulation environments.<n>V-Dreamer employs a novel generative pipeline that constructs physically grounded 3D scenes.<n>We show that our policies robustly generalize to unseen objects in simulation and achieve effective sim-to-real transfer.
  arXiv  Detail & Related papers  (2026-03-19T12:00:04Z)
• URDF-Anything+: Autoregressive Articulated 3D Models Generation for Physical Simulation [45.4820195450296]
  We present an end-to-end autoregressive framework that directly generates executable articulated object models from visual observations.<n>Building on this capability, we enable a new Real-Follow-Sim paradigm, where high-fidelity digital twins constructed from visual observations allow policies trained in simulation to transfer to real robots without online adaptation.
  arXiv  Detail & Related papers  (2026-03-14T16:23:44Z)
• Beyond Pixel Histories: World Models with Persistent 3D State [50.4601060508243]
  PERSIST is a new paradigm of world model which simulates the evolution of a latent 3D scene.<n>We show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods.
  arXiv  Detail & Related papers  (2026-03-03T19:58:31Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• PAT3D: Physics-Augmented Text-to-3D Scene Generation [47.18949891825537]
  PAT3D generates 3D objects, infers their spatial relations, and organizes them into a hierarchical scene tree.<n>A differentiable rigid-body simulator ensures realistic object interactions under gravity.<n>Experiments demonstrate that PAT3D substantially outperforms prior approaches in physical plausibility, semantic consistency, and visual quality.
  arXiv  Detail & Related papers  (2025-11-26T23:23:58Z)
• Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets [63.67760219308476]
  We present Seed3D 1.0, a foundation model that generates simulation-ready 3D assets from single images.<n>Unlike existing 3D generation models, our system produces assets with accurate geometry, well-aligned textures, and realistic physically-based materials.
  arXiv  Detail & Related papers  (2025-10-22T18:16:32Z)
• HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video [25.898073594115413]
  HoloScene is a novel interactive 3D reconstruction framework.<n>It encodes object geometry, appearance, and physical properties alongside hierarchical and inter-object relationships.<n>The resulting digital twins exhibit complete and precise geometry, physical stability, and realistic rendering from novel viewpoints.
  arXiv  Detail & Related papers  (2025-10-07T04:12:18Z)
• EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence [8.987157387248317]
  EmbodiedGen is a foundational platform for interactive 3D world generation.<n>It enables the scalable generation of high-quality, controllable and photorealistic 3D assets at low cost.
  arXiv  Detail & Related papers  (2025-06-12T11:43:50Z)
• R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation [78.26308457952636]
  This paper introduces R3D2, a lightweight, one-step diffusion model designed to overcome limitations in autonomous driving simulation.<n>It enables realistic insertion of complete 3D assets into existing scenes by generating plausible rendering effects-such as shadows and consistent lighting-in real time.<n>We show that R3D2 significantly enhances the realism of inserted assets, enabling use-cases like text-to-3D asset insertion and cross-scene/dataset object transfer.
  arXiv  Detail & Related papers  (2025-06-09T14:50:19Z)
• Is Single-View Mesh Reconstruction Ready for Robotics? [78.14584238127338]
  We evaluate single-view mesh reconstruction models for their potential in enabling instant digital twin creation for real-time planning and dynamics prediction using physics simulators for robotic manipulation.<n>Our findings highlight critical gaps between computer vision advances and robotics needs, guiding future research at this intersection.
  arXiv  Detail & Related papers  (2025-05-23T14:35:56Z)
• SIGHT: Synthesizing Image-Text Conditioned and Geometry-Guided 3D Hand-Object Trajectories [124.24041272390954]
  Modeling hand-object interaction priors holds significant potential to advance robotic and embodied AI systems.<n>We introduce SIGHT, a novel task focused on generating realistic and physically plausible 3D hand-object interaction trajectories from a single image.<n>We propose SIGHT-Fusion, a novel diffusion-based image-text conditioned generative model that tackles this task by retrieving the most similar 3D object mesh from a database.
  arXiv  Detail & Related papers  (2025-03-28T20:53:20Z)
• Generative AI Framework for 3D Object Generation in Augmented Reality [0.0]
  This thesis integrates state-of-the-art generative AI models for real-time creation of 3D objects in augmented reality (AR) environments.<n>The framework demonstrates applications across industries such as gaming, education, retail, and interior design.<n>A significant contribution is democratizing 3D model creation, making advanced AI tools accessible to a broader audience.
  arXiv  Detail & Related papers  (2025-02-21T17:01:48Z)
• Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication [50.541882834405946]
  We introduce Atlas3D, an automatic and easy-to-implement text-to-3D method. Our approach combines a novel differentiable simulation-based loss function with physically inspired regularization. We verify Atlas3D's efficacy through extensive generation tasks and validate the resulting 3D models in both simulated and real-world environments.
  arXiv  Detail & Related papers  (2024-05-28T18:33:18Z)
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation [75.0278287071591]
  ThreeDWorld (TDW) is a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science.
  arXiv  Detail & Related papers  (2020-07-09T17:33:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.