Hybrid Mesh-neural Representation for 3D Transparent Object Reconstruction

• URL: http://arxiv.org/abs/2203.12613v3
• Date: Wed, 29 Mar 2023 07:34:40 GMT
• Title: Hybrid Mesh-neural Representation for 3D Transparent Object Reconstruction
• Authors: Jiamin Xu, Zihan Zhu, Hujun Bao, Weiwei Xu
• Abstract summary: We propose a novel method to reconstruct the 3D shapes of transparent objects using hand-held captured images under natural light conditions. It combines the advantage of explicit mesh and multi-layer perceptron (MLP) network, a hybrid representation, to simplify the capture used in recent contributions.
• Score: 30.66452291775852
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a novel method to reconstruct the 3D shapes of transparent objects using hand-held captured images under natural light conditions. It combines the advantage of explicit mesh and multi-layer perceptron (MLP) network, a hybrid representation, to simplify the capture setting used in recent contributions. After obtaining an initial shape through the multi-view silhouettes, we introduce surface-based local MLPs to encode the vertex displacement field (VDF) for the reconstruction of surface details. The design of local MLPs allows to represent the VDF in a piece-wise manner using two layer MLP networks, which is beneficial to the optimization algorithm. Defining local MLPs on the surface instead of the volume also reduces the searching space. Such a hybrid representation enables us to relax the ray-pixel correspondences that represent the light path constraint to our designed ray-cell correspondences, which significantly simplifies the implementation of single-image based environment matting algorithm. We evaluate our representation and reconstruction algorithm on several transparent objects with ground truth models. Our experiments show that our method can produce high-quality reconstruction results superior to state-of-the-art methods using a simplified data acquisition setup.

Related papers

• RISE-SDF: a Relightable Information-Shared Signed Distance Field for Glossy Object Inverse Rendering [26.988572852463815]
  In this paper, we propose a novel end-to-end relightable neural inverse rendering system. Our algorithm achieves state-of-the-art performance in inverse rendering and relighting. Our experiments demonstrate that our algorithm achieves state-of-the-art performance in inverse rendering and relighting.
  arXiv  Detail & Related papers  (2024-09-30T09:42:10Z)
• PBIR-NIE: Glossy Object Capture under Non-Distant Lighting [30.325872237020395]
  Glossy objects present a significant challenge for 3D reconstruction from multi-view input images under natural lighting. We introduce PBIR-NIE, an inverse rendering framework designed to holistically capture the geometry, material attributes, and surrounding illumination of such objects.
  arXiv  Detail & Related papers  (2024-08-13T13:26:24Z)
• Anti-Aliased Neural Implicit Surfaces with Encoding Level of Detail [54.03399077258403]
  We present LoD-NeuS, an efficient neural representation for high-frequency geometry detail recovery and anti-aliased novel view rendering. Our representation aggregates space features from a multi-convolved featurization within a conical frustum along a ray.
  arXiv  Detail & Related papers  (2023-09-19T05:44:00Z)
• Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering [84.37776381343662]
  Mip-NeRF proposes a multiscale representation as a conical frustum to encode scale information. We propose mip voxel grids (Mip-VoG), an explicit multiscale representation for real-time anti-aliasing rendering. Our approach is the first to offer multiscale training and real-time anti-aliasing rendering simultaneously.
  arXiv  Detail & Related papers  (2023-04-20T04:05:22Z)
• $PC^2$: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction [97.06927852165464]
  Reconstructing the 3D shape of an object from a single RGB image is a long-standing and highly challenging problem in computer vision. We propose a novel method for single-image 3D reconstruction which generates a sparse point cloud via a conditional denoising diffusion process.
  arXiv  Detail & Related papers  (2023-02-21T13:37:07Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• Implicit Neural Deformation for Multi-View Face Reconstruction [43.88676778013593]
  We present a new method for 3D face reconstruction from multi-view RGB images. Unlike previous methods which are built upon 3D morphable models, our method leverages an implicit representation to encode rich geometric features. Our experimental results on several benchmark datasets demonstrate that our approach outperforms alternative baselines and achieves superior face reconstruction results compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-12-05T07:02:53Z)
• Gradient-SDF: A Semi-Implicit Surface Representation for 3D Reconstruction [53.315347543761426]
  Gradient-SDF is a novel representation for 3D geometry that combines the advantages of implict and explicit representations. By storing at every voxel both the signed distance field as well as its gradient vector field, we enhance the capability of implicit representations. We show that (1) the Gradient-SDF allows us to perform direct SDF tracking from depth images, using efficient storage schemes like hash maps, and that (2) the Gradient-SDF representation enables us to perform photometric bundle adjustment directly in a voxel representation.
  arXiv  Detail & Related papers  (2021-11-26T18:33:14Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• Ladybird: Quasi-Monte Carlo Sampling for Deep Implicit Field Based 3D Reconstruction with Symmetry [12.511526058118143]
  We propose a sampling scheme that theoretically encourages generalization and results in fast convergence for SGD-based optimization algorithms. Based on the reflective symmetry of an object, we propose a feature fusion method that alleviates issues due to self-occlusions. Our proposed system Ladybird is able to create high quality 3D object reconstructions from a single input image.
  arXiv  Detail & Related papers  (2020-07-27T09:17:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.