GS-Agent: Creating 4D Physical Worlds With Generative Simulation

• URL: http://arxiv.org/abs/2607.21522v1
• Date: Thu, 23 Jul 2026 17:04:36 GMT
• Title: GS-Agent: Creating 4D Physical Worlds With Generative Simulation
• Abstract summary: We present GS-Agent, an end-to-end multi-agent framework that integrates physics engines in the loop to generate 4D physical worlds from natural language.<n>Inspired by how humans build 4D worlds, GS-Agent decomposes the task into entity management, covering 3D asset curation, material tuning, placement, and motion control, and rendering configuration.<n>We envision GS-Agent as a foundation for a new paradigm in 4D world generation, empowering creative content creation and physical AI.
• Score: 41.82781441252822
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fascinating and challenging. Traditional computer graphics methods rely on manual creation, requiring extensive human effort to fine-tune materials, motions, and visual fidelity. Recent advances in generative foundation models have sparked interest in learning to generate such 4D worlds from large-scale data; however, existing methods still struggle to ensure physical plausibility and controllability. In this work, we take a different path by leveraging foundation models to construct an agentic system that emulates how humans traditionally create 4D worlds, yet automates the entire process. We present GS-Agent, an end-to-end multi-agent framework that integrates physics engines in the loop to generate realistic, dynamic, and controllable 4D physical worlds from natural language. Inspired by how humans build 4D worlds, GS-Agent decomposes the task into entity management, covering 3D asset curation, material tuning, placement, and motion control, and rendering configuration, including camera and lighting manipulation. Multiple agents with distinct expertise interact with the physics engine via code, seek multimodal feedback, and collaborate to iteratively construct 4D worlds that align with the given descriptions. Experimental results show that GS-Agent effectively converts natural language into diverse and physically plausible 4D worlds exhibiting rich interactions among liquids, deformable objects, and rigid bodies, while achieving cinematic camera and lighting control. We envision GS-Agent as a foundation for a new paradigm in 4D world generation, empowering creative content creation and physical AI. Project page at https://umass-embodied-agi.github.io/gs-agent/

Related papers

• SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation [50.175987640001644]
  We present SimWorlds: a multi-agent framework that produces dynamic, editable 4D scenes from text.<n>We also introduce 4DBuildBench, a benchmark for assessing both visual fidelity and physical consistency of the procedural dynamic 3D scenes.
  arXiv  Detail & Related papers  (2026-07-02T06:29:18Z)
• CP4D: Compositional Physics-aware 4D Scene Generation [37.130324925592795]
  We present CP4D, a novel paradigm for 4D scene synthesis with faithful adherence to complex physical dynamics.<n>We show that CP4D can generate explorable and interactive 4D scenes with high visual fidelity, strong physical plausibility, and fine-grained controllability, outperforming significantly existing methods.
  arXiv  Detail & Related papers  (2026-06-08T08:23:12Z)
• GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors [113.71148915419246]
  We present GRAIL, a digital generation pipeline that composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot.<n>GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction.<n>We retarget the recovered motions to a humanoid robot and train complementary task-general morphology trackers.<n>GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting
  arXiv  Detail & Related papers  (2026-06-03T17:57:45Z)
• Code2Worlds: Empowering Coding LLMs for 4D World Generation [14.349376975089607]
  We introduce Code2Worlds, a framework that formulates 4D generation as language-to-simulation code generation.<n>We propose a dual-stream architecture that disentangles retrieval-augmented object generation from hierarchical environmental orchestration.<n>We establish a physics-aware closed-loop mechanism in which a PostProcess Agent scripts dynamics, coupled with a VLM-Motion Critic that performs self-reflection to iteratively refine simulation code.
  arXiv  Detail & Related papers  (2026-02-12T09:34:28Z)
• WorldReel: 4D Video Generation with Consistent Geometry and Motion Modeling [63.37476802589492]
  We present WorldReel, a 4D video that is native-temporally consistent.<n>WorldReel produces frames together with 4D scene representations, including pointmaps, camera trajectory, and dense flow.<n>We believe that WorldReel brings video generation closer to 4D-consistent world modeling.
  arXiv  Detail & Related papers  (2025-12-08T18:54:12Z)
• WorldGen: From Text to Traversable and Interactive 3D Worlds [87.95088818329403]
  We introduce WorldGen, a system that enables the automatic creation of large-scale, interactive 3D worlds directly from text prompts.<n>Our approach transforms natural language descriptions into fully textured environments that can be immediately explored or edited within standard game engines.<n>This work represents a step towards accessible, generative world-building at scale, advancing the frontier of 3D generative AI for applications in gaming, simulation, and immersive social environments.
  arXiv  Detail & Related papers  (2025-11-20T22:13:18Z)
• LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generation [35.4193352348583]
  We propose a simple yet effective 3D world generation framework that streamlines the industrial production pipeline of 3D environments.<n>LatticeWorld creates large-scale 3D interactive worlds with dynamic agents, featuring competitive multi-agent interaction.<n>LatticeWorld achieves over a $90times$ increase in industrial production efficiency.
  arXiv  Detail & Related papers  (2025-09-05T17:22:33Z)
• WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agents [67.31920821192323]
  We introduce WorldCraft, a system where large language model (LLM) agents leverage procedural generation to create scenes populated with objects.<n>In our framework, a coordinator agent manages the overall process and works with two specialized LLM agents to complete the scene creation.<n>Our pipeline incorporates a trajectory control agent, allowing users to animate the scene and operate the camera through natural language interactions.
  arXiv  Detail & Related papers  (2025-02-21T17:18:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.