Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes
- URL: http://arxiv.org/abs/2211.10206v4
- Date: Tue, 21 Mar 2023 07:50:56 GMT
- Title: Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes
- Authors: Zhen Li, Lingli Wang, Mofang Cheng, Cihui Pan, Jiaqi Yang
- Abstract summary: We present a efficient multi-view inverse rendering method for large-scale real-world indoor scenes.
The proposed method outperforms the state-of-the-art quantitatively and qualitatively.
It enables physically-reasonable mixed-reality applications such as material editing, editable novel view synthesis and relighting.
- Score: 5.9870673031762545
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: We present a efficient multi-view inverse rendering method for large-scale
real-world indoor scenes that reconstructs global illumination and
physically-reasonable SVBRDFs. Unlike previous representations, where the
global illumination of large scenes is simplified as multiple environment maps,
we propose a compact representation called Texture-based Lighting (TBL). It
consists of 3D mesh and HDR textures, and efficiently models direct and
infinite-bounce indirect lighting of the entire large scene. Based on TBL, we
further propose a hybrid lighting representation with precomputed irradiance,
which significantly improves the efficiency and alleviates the rendering noise
in the material optimization. To physically disentangle the ambiguity between
materials, we propose a three-stage material optimization strategy based on the
priors of semantic segmentation and room segmentation. Extensive experiments
show that the proposed method outperforms the state-of-the-art quantitatively
and qualitatively, and enables physically-reasonable mixed-reality applications
such as material editing, editable novel view synthesis and relighting. The
project page is at https://lzleejean.github.io/TexIR.
Related papers
- GS-Phong: Meta-Learned 3D Gaussians for Relightable Novel View Synthesis [63.5925701087252]
We propose a novel method for representing a scene illuminated by a point light using a set of relightable 3D Gaussian points.
Inspired by the Blinn-Phong model, our approach decomposes the scene into ambient, diffuse, and specular components.
To facilitate the decomposition of geometric information independent of lighting conditions, we introduce a novel bilevel optimization-based meta-learning framework.
arXiv Detail & Related papers (2024-05-31T13:48:54Z) - IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images [32.83096814910201]
We present a method that recovers the physically based material properties and lighting of a scene from multi-view, low-dynamic-range (LDR) images.
Our method outperforms existing methods taking LDR images as input, and allows for highly realistic relighting and object insertion.
arXiv Detail & Related papers (2024-01-23T18:59:56Z) - Holistic Inverse Rendering of Complex Facade via Aerial 3D Scanning [38.72679977945778]
We use multi-view aerial images to reconstruct the geometry, lighting, and material of facades using neural signed distance fields (SDFs)
The experiment demonstrates the superior quality of our method on facade holistic inverse rendering, novel view synthesis, and scene editing compared to state-of-the-art baselines.
arXiv Detail & Related papers (2023-11-20T15:03:56Z) - Differentiable Blocks World: Qualitative 3D Decomposition by Rendering
Primitives [70.32817882783608]
We present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives.
Unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images.
We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points.
arXiv Detail & Related papers (2023-07-11T17:58:31Z) - Spatiotemporally Consistent HDR Indoor Lighting Estimation [66.26786775252592]
We propose a physically-motivated deep learning framework to solve the indoor lighting estimation problem.
Given a single LDR image with a depth map, our method predicts spatially consistent lighting at any given image position.
Our framework achieves photorealistic lighting prediction with higher quality compared to state-of-the-art single-image or video-based methods.
arXiv Detail & Related papers (2023-05-07T20:36:29Z) - NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field
Indirect Illumination [48.42173911185454]
Inverse rendering methods aim to estimate geometry, materials and illumination from multi-view RGB images.
We propose an end-to-end inverse rendering pipeline that decomposes materials and illumination from multi-view images.
arXiv Detail & Related papers (2023-03-29T12:05:19Z) - NeILF: Neural Incident Light Field for Physically-based Material
Estimation [31.230609753253713]
We present a differentiable rendering framework for material and lighting estimation from multi-view images and a reconstructed geometry.
In the framework, we represent scene lightings as the Neural Incident Light Field (NeILF) and material properties as the surface BRDF modelled by multi-layer perceptrons.
arXiv Detail & Related papers (2022-03-14T15:23:04Z) - Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
We present an efficient method for joint optimization of materials and lighting from multi-view image observations.
We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
arXiv Detail & Related papers (2021-11-24T13:58:20Z) - DIB-R++: Learning to Predict Lighting and Material with a Hybrid
Differentiable Renderer [78.91753256634453]
We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables.
In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing.
Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
arXiv Detail & Related papers (2021-10-30T01:59:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.