HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video

• URL: http://arxiv.org/abs/2510.05560v1
• Date: Tue, 07 Oct 2025 04:12:18 GMT
• Title: HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video
• Abstract summary: HoloScene is a novel interactive 3D reconstruction framework.<n>It encodes object geometry, appearance, and physical properties alongside hierarchical and inter-object relationships.<n>The resulting digital twins exhibit complete and precise geometry, physical stability, and realistic rendering from novel viewpoints.
• Score: 25.898073594115413
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Digitizing the physical world into accurate simulation-ready virtual environments offers significant opportunities in a variety of fields such as augmented and virtual reality, gaming, and robotics. However, current 3D reconstruction and scene-understanding methods commonly fall short in one or more critical aspects, such as geometry completeness, object interactivity, physical plausibility, photorealistic rendering, or realistic physical properties for reliable dynamic simulation. To address these limitations, we introduce HoloScene, a novel interactive 3D reconstruction framework that simultaneously achieves these requirements. HoloScene leverages a comprehensive interactive scene-graph representation, encoding object geometry, appearance, and physical properties alongside hierarchical and inter-object relationships. Reconstruction is formulated as an energy-based optimization problem, integrating observational data, physical constraints, and generative priors into a unified, coherent objective. Optimization is efficiently performed via a hybrid approach combining sampling-based exploration with gradient-based refinement. The resulting digital twins exhibit complete and precise geometry, physical stability, and realistic rendering from novel viewpoints. Evaluations conducted on multiple benchmark datasets demonstrate superior performance, while practical use-cases in interactive gaming and real-time digital-twin manipulation illustrate HoloScene's broad applicability and effectiveness. Project page: https://xiahongchi.github.io/HoloScene.

Related papers

• NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation [37.57654523550713]
  NeoWorld-Pro is a framework that reformulates monocular scene reconstruction as procedural programming for interactive 3D environments.<n>A physics-in-the-loop mechanism then iteratively refines the generated programs by validating their execution in a physics engine.
  arXiv  Detail & Related papers  (2026-08-25T08:19:39Z)
• TelePhysics: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interaction [51.01447538245441]
  We propose TelePhysics, a training-free framework that converts a single image into a physically consistent and controllable video.<n>By representing the full scene geometry in a unified spatial coordinate system, TelePhysics resolves object penetration and alignment ambiguity.<n> Experimental results demonstrate that TelePhysics substantially outperforms prior methods in physical fidelity, spatial coherence, and controllability.
  arXiv  Detail & Related papers  (2026-05-19T08:16:44Z)
• 3D Generation for Embodied AI and Robotic Simulation: A Survey [19.441445845534158]
  Embodied AI and robotic systems increasingly depend on scalable, diverse, and physically grounded 3D content for simulation-based training and real-world deployment.<n>This survey reviews 3D generation for embodied AI and organizes the literature around three roles that 3D generation plays in embodied systems.<n>In Data Generator, 3D generation produces simulation-ready objects and assets, including articulated, physically grounded, and deformable content for downstream interaction.<n>In Simulation Environments, it constructs interactive and task-oriented worlds, spanning structure-aware, controllable, and agentic scene generation.<n>In Sim2Real Bridge, it supports digital twin reconstruction
  arXiv  Detail & Related papers  (2026-04-29T10:17:55Z)
• GS-Playground: A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning [55.272458304179025]
  GS-Playground is a multi-modal simulation framework designed to accelerate end-to-end perceptual learning.<n>Our system achieves a breakthrough throughput of 104 FPS at 640x480 resolution, significantly lowering the barrier for large-scale visual RL.
  arXiv  Detail & Related papers  (2026-04-28T10:05:39Z)
• From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation [11.382656265384854]
  We present a generative framework that establishes a generative real-to-sim mapping from real-world panoramas to high-fidelity simulation scenes.<n>Combined with high-quality physics engines and realistic assets, the generated scenes support interactive manipulation tasks.<n>Experiments demonstrate a strong sim-to-real correlation validating our platform's fidelity, and show that extensively scaling up data generation leads to significantly better generalization to unseen scene and object variations.
  arXiv  Detail & Related papers  (2026-04-17T08:06:26Z)
• SimRecon: SimReady Compositional Scene Reconstruction from Real Videos [32.616029685189744]
  Compositional scene reconstruction seeks to create object-centric representations rather than holistic scenes from real-world videos.<n>We propose SimRecon, a framework that realizes a "Perception-Generation-Simulation" pipeline towards cluttered scene reconstruction.
  arXiv  Detail & Related papers  (2026-03-02T17:51:45Z)
• Real-to-Sim for Highly Cluttered Environments via Physics-Consistent Inter-Object Reasoning [11.05191602561784]
  Reconstructing physically valid 3D scenes from single-view observations is a prerequisite for bridging the gap between visual perception and robotic control.<n>We propose a novel physics-constrained Real-to-Sim pipeline that reconstructs physically consistent 3D scenes from single-view RGB-D data.
  arXiv  Detail & Related papers  (2026-02-13T05:24:58Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• GS-Verse: Mesh-based Gaussian Splatting for Physics-aware Interaction in Virtual Reality [1.2271175670122132]
  Current techniques for physically manipulating 3D content within Virtual Reality (VR) often face significant limitations.<n>We introduce GS-Verse, a novel method designed to overcome these challenges by directly integrating an object's mesh with a Gaussian Splatting representation.<n>Our system is designed to be physics-engine-agnostic, granting developers robust deployment flexibility.
  arXiv  Detail & Related papers  (2025-10-13T19:36:47Z)
• Video Perception Models for 3D Scene Synthesis [109.5543506037003]
  VIPScene is a novel framework that exploits the encoded commonsense knowledge of the 3D physical world in video generation models.<n>VIPScene seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object in a scene.
  arXiv  Detail & Related papers  (2025-06-25T16:40:17Z)
• HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation [50.206100327643284]
  HiScene is a novel hierarchical framework that bridges the gap between 2D image generation and 3D object generation.<n>We generate 3D content that aligns with 2D representations while maintaining compositional structure.
  arXiv  Detail & Related papers  (2025-04-17T16:33:39Z)
• PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos [21.441062722848265]
  PhysTwin is a novel framework that uses sparse videos of dynamic objects under interaction to produce a photo- and physically realistic, real-time interactive replica.<n>Our approach centers on two key components: (1) a physics-informed representation that combines spring-mass models for realistic physical simulation, and generative shape models for geometry, and Gaussian splats for rendering.<n>Our method integrates an inverse physics framework with visual perception cues, enabling high-fidelity reconstruction even from partial, occluded, and limited viewpoints.
  arXiv  Detail & Related papers  (2025-03-23T07:49:19Z)
• 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives [115.67081491747943]
  Dynamic 3D scene representation and novel view synthesis are crucial for enabling AR/VR and metaverse applications.<n>We reformulate the reconstruction of a time-varying 3D scene as approximating its underlying 4D volume.<n>We derive several compact variants that effectively reduce the memory footprint to address its storage bottleneck.
  arXiv  Detail & Related papers  (2024-12-30T05:30:26Z)
• Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splatting [32.846428862045634]
  We present Sim Anything, a physics-based approach that endows static 3D objects with interactive dynamics.<n>Inspired by human visual reasoning, we propose MLLM-based Physical Property Perception.<n>We also simulate objects in an open-world scene with particles sampled via the Physical-Geometric Adaptive Sampling.
  arXiv  Detail & Related papers  (2024-11-19T12:52:21Z)
• Zero-Shot Multi-Object Scene Completion [59.325611678171974]
  We present a 3D scene completion method that recovers the complete geometry of multiple unseen objects in complex scenes from a single RGB-D image. Our method outperforms the current state-of-the-art on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-03-21T17:59:59Z)
• HSPACE: Synthetic Parametric Humans Animated in Complex Environments [67.8628917474705]
  We build a large-scale photo-realistic dataset, Human-SPACE, of animated humans placed in complex indoor and outdoor environments. We combine a hundred diverse individuals of varying ages, gender, proportions, and ethnicity, with hundreds of motions and scenes, in order to generate an initial dataset of over 1 million frames. Assets are generated automatically, at scale, and are compatible with existing real time rendering and game engines.
  arXiv  Detail & Related papers  (2021-12-23T22:27:55Z)
• GeoSim: Photorealistic Image Simulation with Geometry-Aware Composition [81.24107630746508]
  We present GeoSim, a geometry-aware image composition process that synthesizes novel urban driving scenes. We first build a diverse bank of 3D objects with both realistic geometry and appearance from sensor data. The resulting synthetic images are photorealistic, traffic-aware, and geometrically consistent, allowing image simulation to scale to complex use cases.
  arXiv  Detail & Related papers  (2021-01-16T23:00:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.