SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds

• URL: http://arxiv.org/abs/2512.01078v1
• Date: Sun, 30 Nov 2025 20:58:13 GMT
• Title: SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds
• Authors: Jiawei Ren, Yan Zhuang, Xiaokang Ye, Lingjun Mao, Xuhong He, Jianzhi Shen, Mrinaal Dogra, Yiming Liang, Ruixuan Zhang, Tianai Yue, Yiqing Yang, Eric Liu, Ryan Wu, Kevin Benavente, Rajiv Mandya Nagaraju, Muhammad Faayez, Xiyan Zhang, Dhruv Vivek Sharma, Xianrui Zhong, Ziqiao Ma, Tianmin Shu, Zhiting Hu, Lianhui Qin,
• Abstract summary: We introduce SimWorld, a new simulator built on Unreal Engine 5 designed for developing and evaluating AI agents.<n>SimWorld offers realistic, open-ended world simulation, including accurate physical and social dynamics and language-driven procedural environment generation.<n>We demonstrate SimWorld by deploying LLM agents on long-horizon multi-agent delivery tasks involving strategic cooperation and competition.
• Score: 31.504258822495768
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While LLM/VLM-powered AI agents have advanced rapidly in math, coding, and computer use, their applications in complex physical and social environments remain challenging. Building agents that can survive and thrive in the real world (for example, by autonomously earning income or running a business) requires massive-scale interaction, reasoning, training, and evaluation across diverse embodied scenarios. However, existing world simulators for such development fall short: they often rely on limited hand-crafted environments, simulate simplified game-like physics and social rules, and lack native support for LLM/VLM agents. We introduce SimWorld, a new simulator built on Unreal Engine 5, designed for developing and evaluating LLM/VLM agents in rich, real-world-like settings. SimWorld offers three core capabilities: (1) realistic, open-ended world simulation, including accurate physical and social dynamics and language-driven procedural environment generation; (2) a rich interface for LLM/VLM agents, with multimodal world inputs and open-vocabulary actions at varying levels of abstraction; and (3) diverse and extensible physical and social reasoning scenarios that are easily customizable by users. We demonstrate SimWorld by deploying frontier LLM agents (e.g., GPT-4o, Gemini-2.5-Flash, Claude-3.5, and DeepSeek-Prover-V2) on long-horizon multi-agent delivery tasks involving strategic cooperation and competition. The results reveal distinct reasoning patterns and limitations across models. We open-source SimWorld and hope it becomes a foundational platform for advancing real-world agent intelligence across disciplines: https://simworld.org.

Related papers

• TongSIM: A General Platform for Simulating Intelligent Machines [59.27575233453533]
  Embodied intelligence focuses on training agents within realistic simulated environments.<n>TongSIM is a high-fidelity, general-purpose platform for training and evaluating embodied agents.
  arXiv  Detail & Related papers  (2025-12-23T10:00:43Z)
• SIMA 2: A Generalist Embodied Agent for Virtual Worlds [87.15489342016714]
  We introduce SIMA 2, a generalist embodied agent that understands and acts in a wide variety of 3D virtual worlds.<n>Built upon a Gemini foundation model, SIMA 2 represents a significant step toward active, goal-directed interaction.
  arXiv  Detail & Related papers  (2025-12-04T13:46:11Z)
• Dyna-Mind: Learning to Simulate from Experience for Better AI Agents [62.21219817256246]
  We argue that current AI agents need ''vicarious trial and error'' - the capacity to mentally simulate alternative futures before acting.<n>We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM agents to integrate such simulation into their reasoning.
  arXiv  Detail & Related papers  (2025-10-10T17:30:18Z)
• SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments [4.966661313606916]
  This paper presents SimPRIVE, a simulation framework for physical robot interaction with virtual environments.<n>Using SimPRIVE, any physical mobile robot running on ROS 2 can easily be configured to move its digital twin in a virtual world built with the Unreal Engine 5 graphic engine.<n>The framework has been validated by testing a reinforcement learning agent trained for obstacle avoidance on an AgileX Scout Mini rover.
  arXiv  Detail & Related papers  (2025-04-30T09:22:55Z)
• Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation [62.5805866419814]
  Vid2Sim is a novel framework that bridges the sim2real gap through a scalable and cost-efficient real2sim pipeline for neural 3D scene reconstruction and simulation.<n>Experiments demonstrate that Vid2Sim significantly improves the performance of urban navigation in the digital twins and real world by 31.2% and 68.3% in success rate.
  arXiv  Detail & Related papers  (2025-01-12T03:01:15Z)
• MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needs [12.987019067098414]
  We propose a multi-agent Minecraft simulator, MineLand, that bridges this gap by introducing three key features: large-scale scalability, limited multimodal senses, and physical needs. Our simulator supports 64 or more agents. Agents have limited visual, auditory, and environmental awareness, forcing them to actively communicate and collaborate to fulfill physical needs like food and resources. Our experiments demonstrate that the simulator, the corresponding benchmark, and the AI agent framework contribute to more ecological and nuanced collective behavior.
  arXiv  Detail & Related papers  (2024-03-28T09:53:41Z)
• V-IRL: Grounding Virtual Intelligence in Real Life [65.87750250364411]
  V-IRL is a platform that enables agents to interact with the real world in a virtual yet realistic environment. Our platform serves as a playground for developing agents that can accomplish various practical tasks.
  arXiv  Detail & Related papers  (2024-02-05T18:59:36Z)
• Learning Interactive Real-World Simulators [96.5991333400566]
  We explore the possibility of learning a universal simulator of real-world interaction through generative modeling. We use the simulator to train both high-level vision-language policies and low-level reinforcement learning policies. Video captioning models can benefit from training with simulated experience, opening up even wider applications.
  arXiv  Detail & Related papers  (2023-10-09T19:42:22Z)
• How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence [16.24370001383615]
  How to adapt driving knowledge learned in simulation to reality becomes a critical issue. Virtual simulation world differs from the real world in many aspects such as lighting, textures, vehicle dynamics, and agents' behaviors. Three categories of approaches to address the reality gap issue: transferring knowledge from simulation to reality (sim2real), learning in digital twins (DTs), and learning by parallel intelligence (PI)
  arXiv  Detail & Related papers  (2023-05-02T09:00:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.