MeTTA: Single-View to 3D Textured Mesh Reconstruction with Test-Time Adaptation

• URL: http://arxiv.org/abs/2408.11465v1
• Date: Wed, 21 Aug 2024 09:35:16 GMT
• Title: MeTTA: Single-View to 3D Textured Mesh Reconstruction with Test-Time Adaptation
• Authors: Kim Yu-Ji, Hyunwoo Ha, Kim Youwang, Jaeheung Surh, Hyowon Ha, Tae-Hyun Oh,
• Abstract summary: We propose MeTTA, a test-time adaptation exploiting generative prior. We design joint optimization of 3D geometry, appearance, and pose to handle OoD cases with only a single view image. MeTTA effectively deals with OoD scenarios at failure cases of existing learning-based 3D reconstruction models.
• Score: 19.15982759396811
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing 3D from a single view image is a long-standing challenge. One of the popular approaches to tackle this problem is learning-based methods, but dealing with the test cases unfamiliar with training data (Out-of-distribution; OoD) introduces an additional challenge. To adapt for unseen samples in test time, we propose MeTTA, a test-time adaptation (TTA) exploiting generative prior. We design joint optimization of 3D geometry, appearance, and pose to handle OoD cases with only a single view image. However, the alignment between the reference image and the 3D shape via the estimated viewpoint could be erroneous, which leads to ambiguity. To address this ambiguity, we carefully design learnable virtual cameras and their self-calibration. In our experiments, we demonstrate that MeTTA effectively deals with OoD scenarios at failure cases of existing learning-based 3D reconstruction models and enables obtaining a realistic appearance with physically based rendering (PBR) textures.

Related papers

• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• 3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective [96.65476492200648]
  This research aims to study a self-supervised 3D clothing reconstruction method. It recovers the geometry shape, and texture of human clothing from a single 2D image.
  arXiv  Detail & Related papers  (2022-04-27T17:46:55Z)
• Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images [82.32776379815712]
  We study the problem of shape generation in 3D mesh representation from a small number of color images with or without camera poses. We adopt to further improve the shape quality by leveraging cross-view information with a graph convolution network. Our model is robust to the quality of the initial mesh and the error of camera pose, and can be combined with a differentiable function for test-time optimization.
  arXiv  Detail & Related papers  (2022-04-21T03:42:31Z)
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
  3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry. This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
  arXiv  Detail & Related papers  (2022-04-09T03:46:18Z)
• Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction [0.0]
  Learning-based approaches for 3D model reconstruction have attracted attention owing to its modern applications. We present a novel sampling algorithm by optimizing the gradient of predicted coordinates based on the variance on the sampling image. We also adopt Frechet Inception Distance (FID) to form a loss function in learning, which helps bridging the gap between rendered images and input images.
  arXiv  Detail & Related papers  (2021-04-29T07:52:54Z)
• MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation [4.202461384355329]
  We propose MonoRUn, a novel 3D object detection framework that learns dense correspondences and geometry in a self-supervised manner. Our proposed approach outperforms current state-of-the-art methods on KITTI benchmark.
  arXiv  Detail & Related papers  (2021-03-23T15:03:08Z)
• Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency [40.56510679634943]
  We propose a self-supervised training architecture by leveraging the multi-view geometry consistency. We design three novel loss functions for multi-view consistency, including the pixel consistency loss, the depth consistency loss, and the facial landmark-based epipolar loss. Our method is accurate and robust, especially under large variations of expressions, poses, and illumination conditions.
  arXiv  Detail & Related papers  (2020-07-24T12:36:09Z)
• From Image Collections to Point Clouds with Self-supervised Shape and Pose Networks [53.71440550507745]
  Reconstructing 3D models from 2D images is one of the fundamental problems in computer vision. We propose a deep learning technique for 3D object reconstruction from a single image. We learn both 3D point cloud reconstruction and pose estimation networks in a self-supervised manner.
  arXiv  Detail & Related papers  (2020-05-05T04:25:16Z)
• Learning Pose-invariant 3D Object Reconstruction from Single-view Images [61.98279201609436]
  In this paper, we explore a more realistic setup of learning 3D shape from only single-view images. The major difficulty lies in insufficient constraints that can be provided by single view images. We propose an effective adversarial domain confusion method to learn pose-disentangled compact shape space.
  arXiv  Detail & Related papers  (2020-04-03T02:47:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.