Triplet: Triangle Patchlet for Mesh-Based Inverse Rendering and Scene Parameters Approximation

• URL: http://arxiv.org/abs/2410.12414v1
• Date: Wed, 16 Oct 2024 09:59:11 GMT
• Title: Triplet: Triangle Patchlet for Mesh-Based Inverse Rendering and Scene Parameters Approximation
• Authors: Jiajie Yang,
• Abstract summary: inverse rendering seeks to derive the physical properties of a scene, including light, geometry, textures, and materials. Meshes, as a traditional representation adopted by many simulation pipeline, still show limited influence in radiance field for inverse rendering. This paper introduces a novel framework called Triangle Patchlet (abbr. Triplet), a mesh-based representation, to comprehensively approximate these parameters.
• Score: 0.0
• License:
• Abstract: Recent advancements in Radiance Fields have significantly improved novel-view synthesis. However, in many real-world applications, the more advanced challenge lies in inverse rendering, which seeks to derive the physical properties of a scene, including light, geometry, textures, and materials. Meshes, as a traditional representation adopted by many simulation pipeline, however, still show limited influence in radiance field for inverse rendering. This paper introduces a novel framework called Triangle Patchlet (abbr. Triplet), a mesh-based representation, to comprehensively approximate these scene parameters. We begin by assembling Triplets with either randomly generated points or sparse points obtained from camera calibration where all faces are treated as an independent element. Next, we simulate the physical interaction of light and optimize the scene parameters using traditional graphics rendering techniques like rasterization and ray tracing, accompanying with density control and propagation. An iterative mesh extracting process is also suggested, where we continue to optimize on geometry and materials with graph-based operation. We also introduce several regulation terms to enable better generalization of materials property. Our framework could precisely estimate the light, materials and geometry with mesh without prior of light, materials and geometry in a unified framework. Experiments demonstrate that our approach can achieve state-of-the-art visual quality while reconstructing high-quality geometry and accurate material properties.

Related papers

• MIRReS: Multi-bounce Inverse Rendering using Reservoir Sampling [17.435649250309904]
  We present MIRReS, a novel two-stage inverse rendering framework. Our method extracts an explicit geometry (triangular mesh) in stage one, and introduces a more realistic physically-based inverse rendering model. Our method effectively estimates indirect illumination, including self-shadowing and internal reflections.
  arXiv  Detail & Related papers  (2024-06-24T07:00:57Z)
• Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes [62.769186261245416]
  We present a novel inverse rendering framework for large urban scenes capable of jointly reconstructing the scene geometry, spatially-varying materials, and HDR lighting from a set of posed RGB images with optional depth. Specifically, we use a neural field to account for the primary rays, and use an explicit mesh (reconstructed from the underlying neural field) for modeling secondary rays that produce higher-order lighting effects such as cast shadows.
  arXiv  Detail & Related papers  (2023-04-06T17:51:54Z)
• Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement [78.48648360358193]
  We present a novel framework that generates textured surface meshes from images. Our approach begins by efficiently initializing the geometry and view-dependency appearance with a NeRF. We jointly refine the appearance with geometry and bake it into texture images for real-time rendering.
  arXiv  Detail & Related papers  (2023-03-03T17:14:44Z)
• NDJIR: Neural Direct and Joint Inverse Rendering for Geometry, Lights, and Materials of Real Object [5.665283675533071]
  We propose neural direct and joint inverse rendering, NDJIR. Our proposed method can decompose semantically well for real object in photogrammetric setting.
  arXiv  Detail & Related papers  (2023-02-02T13:21:03Z)
• Physics-based Indirect Illumination for Inverse Rendering [70.27534648770057]
  We present a physics-based inverse rendering method that learns the illumination, geometry, and materials of a scene from posed multi-view RGB images. As a side product, our physics-based inverse rendering model also facilitates flexible and realistic material editing as well as relighting.
  arXiv  Detail & Related papers  (2022-12-09T07:33:49Z)
• NeROIC: Neural Rendering of Objects from Online Image Collections [42.02832046768925]
  We present a novel method to acquire object representations from online image collections, capturing high-quality geometry and material properties of arbitrary objects. This enables various object-centric rendering applications such as novel-view synthesis, relighting, and harmonized background composition.
  arXiv  Detail & Related papers  (2022-01-07T16:45:15Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer [78.91753256634453]
  We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiables. In this work, we propose DIBR++, a hybrid differentiable which supports these effects by combining specularization and ray-tracing. Compared to more advanced physics-based differentiables, DIBR++ is highly performant due to its compact and expressive model.
  arXiv  Detail & Related papers  (2021-10-30T01:59:39Z)
• PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting [60.75436852495868]
  We present PhySG, an inverse rendering pipeline that reconstructs geometry, materials, and illumination from scratch from RGB input images. We demonstrate, with both synthetic and real data, that our reconstructions not only enable rendering of novel viewpoints, but also physics-based appearance editing of materials and illumination.
  arXiv  Detail & Related papers  (2021-04-01T17:59:02Z)
• Shape From Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF Material from Images via Differentiable Path Tracing [16.975014467319443]
  Differentiable path tracing is an appealing framework as it can reproduce complex appearance effects. We show how to use differentiable ray tracing to refine an initial coarse mesh and per-mesh-facet material representation. We also show how to refine initial reconstructions of real-world objects in unconstrained environments.
  arXiv  Detail & Related papers  (2020-12-06T18:55:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.