PSDR-Room: Single Photo to Scene using Differentiable Rendering

• URL: http://arxiv.org/abs/2307.03244v1
• Date: Thu, 6 Jul 2023 18:17:59 GMT
• Title: PSDR-Room: Single Photo to Scene using Differentiable Rendering
• Authors: Kai Yan, Fujun Luan, Milo\v{S} Ha\v{S}An, Thibault Groueix, Valentin Deschaintre, Shuang Zhao
• Abstract summary: A 3D digital scene contains many components: lights, materials and geometries, interacting to reach the desired appearance. We propose PSDR-Room, a system allowing to optimize lighting as well as the pose and materials of individual objects to match a target image of a room scene, with minimal user input.
• Score: 18.23851486874071
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A 3D digital scene contains many components: lights, materials and geometries, interacting to reach the desired appearance. Staging such a scene is time-consuming and requires both artistic and technical skills. In this work, we propose PSDR-Room, a system allowing to optimize lighting as well as the pose and materials of individual objects to match a target image of a room scene, with minimal user input. To this end, we leverage a recent path-space differentiable rendering approach that provides unbiased gradients of the rendering with respect to geometry, lighting, and procedural materials, allowing us to optimize all of these components using gradient descent to visually match the input photo appearance. We use recent single-image scene understanding methods to initialize the optimization and search for appropriate 3D models and materials. We evaluate our method on real photographs of indoor scenes and demonstrate the editability of the resulting scene components.

Related papers

• Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering [56.68286440268329]
  correct insertion of virtual objects in images of real-world scenes requires a deep understanding of the scene's lighting, geometry and materials. We propose using a personalized large diffusion model as guidance to a physically based inverse rendering process. Our method recovers scene lighting and tone-mapping parameters, allowing the photorealistic composition of arbitrary virtual objects in single frames or videos of indoor or outdoor scenes.
  arXiv  Detail & Related papers  (2024-08-19T05:15:45Z)
• IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images [32.83096814910201]
  We present a method that recovers the physically based material properties and lighting of a scene from multi-view, low-dynamic-range (LDR) images. Our method outperforms existing methods taking LDR images as input, and allows for highly realistic relighting and object insertion.
  arXiv  Detail & Related papers  (2024-01-23T18:59:56Z)
• Scene-Conditional 3D Object Stylization and Composition [30.120066605881448]
  3D generative models have made impressive progress, enabling the generation of almost arbitrary 3D assets from text or image inputs. We propose a framework that allows for the stylization of an existing 3D asset to fit into a given 2D scene, and additionally produce a photorealistic composition as if the asset was placed within the environment.
  arXiv  Detail & Related papers  (2023-12-19T18:50:33Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)
• Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes [62.769186261245416]
  We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows.
  arXiv  Detail & Related papers  (2023-04-06T17:51:54Z)
• MAIR: Multi-view Attention Inverse Rendering with 3D Spatially-Varying Lighting Estimation [13.325800282424598]
  We propose a scene-level inverse rendering framework that uses multi-view images to decompose the scene into geometry, a SVBRDF, and 3D spatially-varying lighting. Our experiments show that the proposed method achieves better performance than single-view-based methods, but also achieves robust performance on unseen real-world scene.
  arXiv  Detail & Related papers  (2023-03-22T08:07:28Z)
• PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes [84.66946637534089]
  PhotoScene is a framework that takes input image(s) of a scene and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D scenes.
  arXiv  Detail & Related papers  (2022-07-02T06:52:44Z)
• Physically-Based Editing of Indoor Scene Lighting from a Single Image [106.60252793395104]
  We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions.
  arXiv  Detail & Related papers  (2022-05-19T06:44:37Z)
• Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
  We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
  arXiv  Detail & Related papers  (2021-09-13T15:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.