Deep-MDS Framework for Recovering the 3D Shape of 2D Landmarks from a
Single Image
- URL: http://arxiv.org/abs/2210.15200v1
- Date: Thu, 27 Oct 2022 06:20:10 GMT
- Title: Deep-MDS Framework for Recovering the 3D Shape of 2D Landmarks from a
Single Image
- Authors: Shima Kamyab, Zohreh Azimifar
- Abstract summary: This paper proposes a framework to recover the 3D shape of 2D landmarks on a human face, in a single input image.
A deep neural network learns the pairwise dissimilarity among 2D landmarks, used by NMDS approach.
- Score: 8.368476827165114
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper, a low parameter deep learning framework utilizing the
Non-metric Multi-Dimensional scaling (NMDS) method, is proposed to recover the
3D shape of 2D landmarks on a human face, in a single input image. Hence, NMDS
approach is used for the first time to establish a mapping from a 2D landmark
space to the corresponding 3D shape space. A deep neural network learns the
pairwise dissimilarity among 2D landmarks, used by NMDS approach, whose
objective is to learn the pairwise 3D Euclidean distance of the corresponding
2D landmarks on the input image. This scheme results in a symmetric
dissimilarity matrix, with the rank larger than 2, leading the NMDS approach
toward appropriately recovering the 3D shape of corresponding 2D landmarks. In
the case of posed images and complex image formation processes like perspective
projection which causes occlusion in the input image, we consider an
autoencoder component in the proposed framework, as an occlusion removal part,
which turns different input views of the human face into a profile view. The
results of a performance evaluation using different synthetic and real-world
human face datasets, including Besel Face Model (BFM), CelebA, CoMA - FLAME,
and CASIA-3D, indicates the comparable performance of the proposed framework,
despite its small number of training parameters, with the related
state-of-the-art and powerful 3D reconstruction methods from the literature, in
terms of efficiency and accuracy.
Related papers
- JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human
Mesh Recovery [84.67823511418334]
This paper presents 3D JOint contrastive learning with TRansformers framework for handling occluded 3D human mesh recovery.
Our method includes an encoder-decoder transformer architecture to fuse 2D and 3D representations for achieving 2D$&$3D aligned results.
arXiv Detail & Related papers (2023-07-31T02:58:58Z) - Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud
Pre-training [65.75399500494343]
Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for 2D and 3D computer vision.
We propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training.
arXiv Detail & Related papers (2023-02-27T17:56:18Z) - RAFaRe: Learning Robust and Accurate Non-parametric 3D Face
Reconstruction from Pseudo 2D&3D Pairs [13.11105614044699]
We propose a robust and accurate non-parametric method for single-view 3D face reconstruction (SVFR)
A large-scale pseudo 2D&3D dataset is created by first rendering the detailed 3D faces, then swapping the face in the wild images with the rendered face.
Our model outperforms previous methods on FaceScape-wild/lab and MICC benchmarks.
arXiv Detail & Related papers (2023-02-10T19:40:26Z) - SSR-2D: Semantic 3D Scene Reconstruction from 2D Images [54.46126685716471]
In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations.
The key idea of our approach is to design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images.
Our method achieves the state-of-the-art performance of semantic scene completion on two large-scale benchmark datasets MatterPort3D and ScanNet.
arXiv Detail & Related papers (2023-02-07T17:47:52Z) - RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects [68.85305626324694]
Ray-marching in Camera Space (RiCS) is a new method to represent the self-occlusions of foreground objects in 3D into a 2D self-occlusion map.
We show that our representation map not only allows us to enhance the image quality but also to model temporally coherent complex shadow effects.
arXiv Detail & Related papers (2022-05-14T05:35:35Z) - Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori.
Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry.
This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
arXiv Detail & Related papers (2022-04-09T03:46:18Z) - Implicit Neural Deformation for Multi-View Face Reconstruction [43.88676778013593]
We present a new method for 3D face reconstruction from multi-view RGB images.
Unlike previous methods which are built upon 3D morphable models, our method leverages an implicit representation to encode rich geometric features.
Our experimental results on several benchmark datasets demonstrate that our approach outperforms alternative baselines and achieves superior face reconstruction results compared to state-of-the-art methods.
arXiv Detail & Related papers (2021-12-05T07:02:53Z) - Learning Geometry-Guided Depth via Projective Modeling for Monocular 3D Object Detection [70.71934539556916]
We learn geometry-guided depth estimation with projective modeling to advance monocular 3D object detection.
Specifically, a principled geometry formula with projective modeling of 2D and 3D depth predictions in the monocular 3D object detection network is devised.
Our method remarkably improves the detection performance of the state-of-the-art monocular-based method without extra data by 2.80% on the moderate test setting.
arXiv Detail & Related papers (2021-07-29T12:30:39Z) - Adaptive 3D Face Reconstruction from a Single Image [45.736818498242016]
We propose a novel joint 2D and 3D optimization method to adaptively reconstruct 3D face shapes from a single image.
Experimental results on multiple datasets demonstrate that our method can generate high-quality reconstruction from a single color image.
arXiv Detail & Related papers (2020-07-08T09:35:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.