SPEAR: A Simulator for Photorealistic Embodied AI Research

• URL: http://arxiv.org/abs/2607.06701v1
• Date: Tue, 07 Jul 2026 18:20:33 GMT
• Title: SPEAR: A Simulator for Photorealistic Embodied AI Research
• Abstract summary: SPEAR is a Python library that can control any Unreal Engine (UE) application via a modular architecture.<n>A single SPEAR instance can render 1920x1080 beauty images directly into a user's NumPy array at 73 frames per second.
• Score: 50.72272540069964
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive simulators have become powerful tools for training embodied agents and generating synthetic visual data, but existing photorealistic simulators suffer from limited generality, programmability, and rendering speed. We address these limitations by introducing SPEAR: A Simulator for Photorealistic Embodied AI Research. At its core, SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions to Python, representing an order-of-magnitude increase in programmable functionality over existing UE-based simulators. Additionally, a single SPEAR instance can render 1920x1080 photorealistic beauty images directly into a user's NumPy array at 73 frames per second - an order of magnitude faster than existing UE plugins - while also providing ground truth image modalities that are not available in any existing UE-based simulator (e.g., a non-diffuse intrinsic image decomposition, material IDs, and physically based shading parameters). Finally, SPEAR introduces an expressive high-level programming model that enables users to specify complex graphs of UE work with arbitrary data dependencies among work items, and to execute these graphs deterministically within a single UE frame. We demonstrate the utility of SPEAR through a diverse collection of example applications: controlling multiple embodied agents with distinct action spaces (e.g., humans, cars, and robots) across several in-the-wild UE projects; rendering photorealistic city-scale environments; manipulating UE's procedural content generation systems; rendering synchronized multi-view images of detailed human faces; coordinating an interactive co-simulation with the MuJoCo physics simulator; and editing scenes with natural language via an AI coding assistant.

Related papers

• Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator [70.34667754804168]
  Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world.<n>We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences.
  arXiv  Detail & Related papers  (2026-07-07T02:42:41Z)
• GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environments [63.222920701596564]
  Training embodied agents in the real world requires skilled operators and expensive hardware.<n> Simulation environments offer a compelling alternative by enabling large-scale, cost-effective data augmentation.<n>Fastly constructing high-fidelity simulation scenes with a minimal sim-to-real gap has become a critical objective in robot learning.
  arXiv  Detail & Related papers  (2026-06-16T05:00:42Z)
• SCOPE: Real-Time Natural Language Camera Agent at the Edge [0.0]
  SCOPE (Simulation and Camera Operations for Perception and Evaluation)<n>We present SCOPE, a modular agent for natural-language openvocry pantiltzoom (PTZ) camera control and visual scene understanding.<n>We release a 536-task benchmark spanning QA, single-language counting, counting, spatial reasoning, descriptions, and optical character recognition in a Blender-based simulation environment.
  arXiv  Detail & Related papers  (2026-06-01T23:07:44Z)
• LychSim: A Controllable and Interactive Simulation Framework for Vision Research [13.971578506119647]
  We present LychSim, a highly controllable and interactive simulation framework built upon Unreal Engine 5.<n>LychSim is built around three key designs: (1) a streamlined Python API that abstracts away underlying engine complexities; (2) a procedural data pipeline capable of generating diverse, high-fidelity environments with varying out-of-distribution (OOD) visual challenges, paired with rich 2D and 3D ground truths; and (3) a native integration of the Model Context Protocol (MCP) that transforms the simulator into a dynamic, closed-loop playground for reasoning agentic LLMs.
  arXiv  Detail & Related papers  (2026-05-12T17:40:38Z)
• GS-Playground: A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning [55.272458304179025]
  GS-Playground is a multi-modal simulation framework designed to accelerate end-to-end perceptual learning.<n>Our system achieves a breakthrough throughput of 104 FPS at 640x480 resolution, significantly lowering the barrier for large-scale visual RL.
  arXiv  Detail & Related papers  (2026-04-28T10:05:39Z)
• Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions [27.247431258140463]
  We present a real-to-sim policy evaluation framework that constructs soft-body digital twins from real-world videos.<n>We validate our approach on representative deformable manipulation tasks, including plush toy packing, rope routing, and T-block pushing.
  arXiv  Detail & Related papers  (2025-11-06T18:52:08Z)
• GaussGym: An open-source real-to-sim framework for learning locomotion from pixels [78.05453137978132]
  We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in within vectorized physics simulators.<n>This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPU.<n>We additionally demonstrate its applicability in a sim-to-real robotics setting.
  arXiv  Detail & Related papers  (2025-10-17T06:34:52Z)
• FIRE-3DV: Framework-Independent Rendering Engine for 3D Graphics using Vulkan [4.226502078427161]
  This paper presents a performance-focused and lightweight rendering engine supporting the modern Vulkan graphics API.<n>Our engine is used to modernize the legacy rendering pipeline of the Asynchronous Multi-Body Framework (AMBF), a dynamic simulation framework.<n>Experiments show that the engine can render a simulated scene with over seven million triangles while maintaining GPU computation times within two milliseconds.
  arXiv  Detail & Related papers  (2024-10-07T14:50:19Z)
• Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents [49.11254369459406]
  ChatSim is the first system that enables editable photo-realistic 3D driving scene simulations via natural language commands with external digital assets. ChatSim employs a novel multi-camera neural field method to generate photo-realistic outcomes.
  arXiv  Detail & Related papers  (2024-02-08T15:26:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.