DRAWER: Digital Reconstruction and Articulation With Environment Realism

• URL: http://arxiv.org/abs/2504.15278v2
• Date: Tue, 22 Apr 2025 05:50:57 GMT
• Title: DRAWER: Digital Reconstruction and Articulation With Environment Realism
• Authors: Hongchi Xia, Entong Su, Marius Memmel, Arhan Jain, Raymond Yu, Numfor Mbiziwo-Tiapo, Ali Farhadi, Abhishek Gupta, Shenlong Wang, Wei-Chiu Ma,
• Abstract summary: We present DRAWER, a novel framework that converts a video of a static indoor scene into a photorealistic and interactive digital environment.<n>We demonstrate the potential of DRAWER by using it to automatically create an interactive game in Unreal Engine and to enable real-to-sim-to-real transfer for robotics applications.
• Score: 42.13130021795826
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Creating virtual digital replicas from real-world data unlocks significant potential across domains like gaming and robotics. In this paper, we present DRAWER, a novel framework that converts a video of a static indoor scene into a photorealistic and interactive digital environment. Our approach centers on two main contributions: (i) a reconstruction module based on a dual scene representation that reconstructs the scene with fine-grained geometric details, and (ii) an articulation module that identifies articulation types and hinge positions, reconstructs simulatable shapes and appearances and integrates them into the scene. The resulting virtual environment is photorealistic, interactive, and runs in real time, with compatibility for game engines and robotic simulation platforms. We demonstrate the potential of DRAWER by using it to automatically create an interactive game in Unreal Engine and to enable real-to-sim-to-real transfer for robotics applications.

Related papers

• SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments [4.966661313606916]
  This paper presents SimPRIVE, a simulation framework for physical robot interaction with virtual environments. Using SimPRIVE, any physical mobile robot running on ROS 2 can easily be configured to move its digital twin in a virtual world built with the Unreal Engine 5 graphic engine. The framework has been validated by testing a reinforcement learning agent trained for obstacle avoidance on an AgileX Scout Mini rover.
  arXiv  Detail & Related papers  (2025-04-30T09:22:55Z)
• PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos [21.441062722848265]
  PhysTwin is a novel framework that uses sparse videos of dynamic objects under interaction to produce a photo- and physically realistic, real-time interactive replica.<n>Our approach centers on two key components: (1) a physics-informed representation that combines spring-mass models for realistic physical simulation, and generative shape models for geometry, and Gaussian splats for rendering.<n>Our method integrates an inverse physics framework with visual perception cues, enabling high-fidelity reconstruction even from partial, occluded, and limited viewpoints.
  arXiv  Detail & Related papers  (2025-03-23T07:49:19Z)
• VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion [25.440573256776133]
  This paper presents a Real-to-Sim-to-Real framework that generates and physically interactive "digital twin" simulation environments for visual navigation and locomotion learning.
  arXiv  Detail & Related papers  (2025-02-03T17:15:05Z)
• URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images [39.0780707100513]
  We present an integrated end-to-end pipeline that generates simulation scenes complete with articulated kinematic and dynamic structures from real-world images. In doing so, our work provides both a pipeline for large-scale generation of simulation environments and an integrated system for training robust robotic control policies.
  arXiv  Detail & Related papers  (2024-05-19T20:01:29Z)
• Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video [23.484070818399]
  Video2Game is a novel approach that automatically converts videos of real-world scenes into realistic and interactive game environments. We show that we can not only produce highly-realistic renderings in real-time, but also build interactive games on top.
  arXiv  Detail & Related papers  (2024-04-15T14:32:32Z)
• Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments [81.38641691636847]
  We rethink the problem of scene reconstruction from an embodied agent's perspective. We reconstruct an interactive scene using RGB-D data stream. This reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models.
  arXiv  Detail & Related papers  (2021-03-30T05:56:58Z)
• GeoSim: Photorealistic Image Simulation with Geometry-Aware Composition [81.24107630746508]
  We present GeoSim, a geometry-aware image composition process that synthesizes novel urban driving scenes. We first build a diverse bank of 3D objects with both realistic geometry and appearance from sensor data. The resulting synthetic images are photorealistic, traffic-aware, and geometrically consistent, allowing image simulation to scale to complex use cases.
  arXiv  Detail & Related papers  (2021-01-16T23:00:33Z)
• OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets [103.54691385842314]
  We propose a novel framework for creating large-scale photorealistic datasets of indoor scenes. Our goal is to make the dataset creation process widely accessible. This enables important applications in inverse rendering, scene understanding and robotics.
  arXiv  Detail & Related papers  (2020-07-25T06:48:47Z)
• ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation [75.0278287071591]
  ThreeDWorld (TDW) is a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science.
  arXiv  Detail & Related papers  (2020-07-09T17:33:27Z)
• Learning to Simulate Dynamic Environments with GameGAN [109.25308647431952]
  In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training.
  arXiv  Detail & Related papers  (2020-05-25T14:10:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.