Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator

• URL: http://arxiv.org/abs/2607.05765v1
• Date: Tue, 07 Jul 2026 02:42:41 GMT
• Title: Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator
• Abstract summary: Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world.<n>We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences.
• Score: 70.34667754804168
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.

Related papers

• ABot-Earth 0.5: Generative 3D Earth Model [30.627822427932273]
  ABot-Earth 0.5 is a generative 3D framework designed to synthesize vast, seamless 3D environments from ubiquitous, geospatially referenced satellite imagery.<n>At inference, it synthesizes novel 3D scenes conditioned solely on satellite imagery at a scalable rate of under 10 minutes per square kilometer.
  arXiv  Detail & Related papers  (2026-06-08T17:58:02Z)
• GS-Playground: A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning [55.272458304179025]
  GS-Playground is a multi-modal simulation framework designed to accelerate end-to-end perceptual learning.<n>Our system achieves a breakthrough throughput of 104 FPS at 640x480 resolution, significantly lowering the barrier for large-scale visual RL.
  arXiv  Detail & Related papers  (2026-04-28T10:05:39Z)
• Habitat-GS: A High-Fidelity Navigation Simulator with Dynamic Gaussian Splatting [83.60803397468139]
  Habitat-GS is a navigation-centric embodied AI simulator that integrates 3D photorealistic scene rendering and drivable gaussian avatars.<n>For dynamic human modeling, we introduce a gaussian avatar module that enables each avatar to simultaneously serve as a photorealistic visual entity and an effective navigation obstacle.
  arXiv  Detail & Related papers  (2026-04-14T11:52:59Z)
• NVSim: Novel View Synthesis Simulator for Large Scale Indoor Navigation [0.39198548406564604]
  We present NVSim, a framework that automatically constructs large-scale, navigable indoor simulators from only common image sequences.<n>We introduce Floor-Aware Gaussian Splatting to ensure a clean, navigable ground plane, and a novel mesh-free traversability checking algorithm.<n>We demonstrate our system's ability to generate valid, large-scale navigation graphs from real-world data.
  arXiv  Detail & Related papers  (2025-10-28T11:57:33Z)
• GaussGym: An open-source real-to-sim framework for learning locomotion from pixels [78.05453137978132]
  We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in within vectorized physics simulators.<n>This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPU.<n>We additionally demonstrate its applicability in a sim-to-real robotics setting.
  arXiv  Detail & Related papers  (2025-10-17T06:34:52Z)
• Simple and Effective Synthesis of Indoor 3D Scenes [78.95697556834536]
  We study the problem of immersive 3D indoor scenes from one or more images. Our aim is to generate high-resolution images and videos from novel viewpoints. We propose an image-to-image GAN that maps directly from reprojections of incomplete point clouds to full high-resolution RGB-D images.
  arXiv  Detail & Related papers  (2022-04-06T17:54:46Z)
• GeoSim: Photorealistic Image Simulation with Geometry-Aware Composition [81.24107630746508]
  We present GeoSim, a geometry-aware image composition process that synthesizes novel urban driving scenes. We first build a diverse bank of 3D objects with both realistic geometry and appearance from sensor data. The resulting synthetic images are photorealistic, traffic-aware, and geometrically consistent, allowing image simulation to scale to complex use cases.
  arXiv  Detail & Related papers  (2021-01-16T23:00:33Z)
• Photorealism in Driving Simulations: Blending Generative Adversarial Image Synthesis with Rendering [0.0]
  We introduce a hybrid generative neural graphics pipeline for improving the visual fidelity of driving simulations. We form 2D semantic images from 3D scenery consisting of simple object models without textures. These semantic images are then converted into photorealistic RGB images with a state-of-the-art Generative Adrial Network (GAN) trained on real-world driving scenes.
  arXiv  Detail & Related papers  (2020-07-31T03:25:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.