DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images

• URL: http://arxiv.org/abs/2007.06127v1
• Date: Sun, 12 Jul 2020 23:13:06 GMT
• Title: DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images
• Abstract summary: We propose a Differentiable Renderer Without Rendering (DRWR) for unsupervised 3D structure learning from 2D images. DRWR relies on a simple but effective loss that evaluates how well the projections of reconstructed 3D point clouds cover the ground truth object silhouette. Our results demonstrate that DRWR state-of-the-art accuracies under widely used benchmarks, outperforming previous methods both qualitatively and quantitatively.
• Score: 70.54067838587692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentiable renderers have been used successfully for unsupervised 3D structure learning from 2D images because they can bridge the gap between 3D and 2D. To optimize 3D shape parameters, current renderers rely on pixel-wise losses between rendered images of 3D reconstructions and ground truth images from corresponding viewpoints. Hence they require interpolation of the recovered 3D structure at each pixel, visibility handling, and optionally evaluating a shading model. In contrast, here we propose a Differentiable Renderer Without Rendering (DRWR) that omits these steps. DRWR only relies on a simple but effective loss that evaluates how well the projections of reconstructed 3D point clouds cover the ground truth object silhouette. Specifically, DRWR employs a smooth silhouette loss to pull the projection of each individual 3D point inside the object silhouette, and a structure-aware repulsion loss to push each pair of projections that fall inside the silhouette far away from each other. Although we omit surface interpolation, visibility handling, and shading, our results demonstrate that DRWR achieves state-of-the-art accuracies under widely used benchmarks, outperforming previous methods both qualitatively and quantitatively. In addition, our training times are significantly lower due to the simplicity of DRWR.

Related papers

• SR3D: Unleashing Single-view 3D Reconstruction for Transparent and Specular Object Grasping [7.222966501323922]
  We propose a training free framework SR3D that enables robotic grasping of transparent and specular objects from a single view observation.<n>Specifically, given single view RGB and depth images, SR3D first uses the external visual models to generate 3D reconstructed object mesh.<n>Then, the key idea is to determine the 3D object's pose and scale to accurately localize the reconstructed object back into its original depth corrupted 3D scene.
  arXiv  Detail & Related papers  (2025-05-30T07:38:46Z)
• Learning A Zero-shot Occupancy Network from Vision Foundation Models via Self-supervised Adaptation [41.98740330990215]
  This work proposes a novel approach that bridges 2D vision foundation models with 3D tasks. We leverage the zero-shot capabilities of vision-language models for image semantics. We project the semantics into 3D space using the reconstructed metric depth, thereby providing 3D supervision.
  arXiv  Detail & Related papers  (2025-03-10T09:54:40Z)
• GaussRender: Learning 3D Occupancy with Gaussian Rendering [86.89653628311565]
  GaussRender is a module that improves 3D occupancy learning by enforcing projective consistency.<n>Our method penalizes 3D configurations that produce inconsistent 2D projections, thereby enforcing a more coherent 3D structure.
  arXiv  Detail & Related papers  (2025-02-07T16:07:51Z)
• Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors [17.544733016978928]
  3D object generation from a single image involves estimating the full 3D geometry and texture of unseen views from an unposed RGB image captured in the wild. Recent advancements in 3D object generation have introduced techniques that reconstruct an object's 3D shape and texture. We propose bridging the gap between 2D and 3D diffusion models to address this limitation.
  arXiv  Detail & Related papers  (2024-10-12T10:14:11Z)
• VCD-Texture: Variance Alignment based 3D-2D Co-Denoising for Text-Guided Texturing [22.39760469467524]
  We propose a Variance texture synthesis to address the modal gap between the 2D and 3D diffusion models. We present an inpainting module to improve details with conflicting regions.
  arXiv  Detail & Related papers  (2024-07-05T12:11:33Z)
• GAN2X: Non-Lambertian Inverse Rendering of Image GANs [85.76426471872855]
  We present GAN2X, a new method for unsupervised inverse rendering that only uses unpaired images for training. Unlike previous Shape-from-GAN approaches that mainly focus on 3D shapes, we take the first attempt to also recover non-Lambertian material properties by exploiting the pseudo paired data generated by a GAN. Experiments demonstrate that GAN2X can accurately decompose 2D images to 3D shape, albedo, and specular properties for different object categories, and achieves the state-of-the-art performance for unsupervised single-view 3D face reconstruction.
  arXiv  Detail & Related papers  (2022-06-18T16:58:49Z)
• 3D-Aware Indoor Scene Synthesis with Depth Priors [62.82867334012399]
  Existing methods fail to model indoor scenes due to the large diversity of room layouts and the objects inside. We argue that indoor scenes do not have a shared intrinsic structure, and hence only using 2D images cannot adequately guide the model with the 3D geometry.
  arXiv  Detail & Related papers  (2022-02-17T09:54:29Z)
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis [163.96778522283967]
  We propose a shading-guided generative implicit model that is able to learn a starkly improved shape representation. An accurate 3D shape should also yield a realistic rendering under different lighting conditions. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2021-10-29T10:53:12Z)
• Style Agnostic 3D Reconstruction via Adversarial Style Transfer [23.304453155586312]
  Reconstructing the 3D geometry of an object from an image is a major challenge in computer vision. We propose an approach that enables a differentiable-based learning of 3D objects from images with backgrounds.
  arXiv  Detail & Related papers  (2021-10-20T21:24:44Z)
• Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching [66.98712589559028]
  We propose an unsupervised approach for 3D point cloud generation with fine structures. Our method can recover fine 3D structures from 2D silhouette images at different resolutions.
  arXiv  Detail & Related papers  (2021-08-08T22:15:31Z)
• Model-based 3D Hand Reconstruction via Self-Supervised Learning [72.0817813032385]
  Reconstructing a 3D hand from a single-view RGB image is challenging due to various hand configurations and depth ambiguity. We propose S2HAND, a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations.
  arXiv  Detail & Related papers  (2021-03-22T10:12:43Z)
• An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering [0.0]
  Differentiable rendering is a very successful technique that applies to a Single-View 3D Reconstruction. Currents use losses based on pixels between a rendered image of some 3D reconstructed object and ground-truth images from given matched viewpoints to optimise parameters of the 3D shape. We propose a novel effective loss function that evaluates how well the projections of reconstructed 3D point clouds cover the ground truth object's silhouette.
  arXiv  Detail & Related papers  (2021-03-05T00:02:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.