SADIR: Shape-Aware Diffusion Models for 3D Image Reconstruction

• URL: http://arxiv.org/abs/2309.03335v2
• Date: Tue, 3 Oct 2023 09:40:35 GMT
• Title: SADIR: Shape-Aware Diffusion Models for 3D Image Reconstruction
• Authors: Nivetha Jayakumar, Tonmoy Hossain, Miaomiao Zhang
• Abstract summary: 3D image reconstruction from a limited number of 2D images has been a long-standing challenge in computer vision and image analysis. We propose a shape-aware network based on diffusion models for 3D image reconstruction, named SADIR, to address these issues.
• Score: 2.2954246824369218
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D image reconstruction from a limited number of 2D images has been a long-standing challenge in computer vision and image analysis. While deep learning-based approaches have achieved impressive performance in this area, existing deep networks often fail to effectively utilize the shape structures of objects presented in images. As a result, the topology of reconstructed objects may not be well preserved, leading to the presence of artifacts such as discontinuities, holes, or mismatched connections between different parts. In this paper, we propose a shape-aware network based on diffusion models for 3D image reconstruction, named SADIR, to address these issues. In contrast to previous methods that primarily rely on spatial correlations of image intensities for 3D reconstruction, our model leverages shape priors learned from the training data to guide the reconstruction process. To achieve this, we develop a joint learning network that simultaneously learns a mean shape under deformation models. Each reconstructed image is then considered as a deformed variant of the mean shape. We validate our model, SADIR, on both brain and cardiac magnetic resonance images (MRIs). Experimental results show that our method outperforms the baselines with lower reconstruction error and better preservation of the shape structure of objects within the images.

Related papers

• 3DMiner: Discovering Shapes from Large-Scale Unannotated Image Datasets [34.610546020800236]
  3DMiner is a pipeline for mining 3D shapes from challenging datasets. Our method is capable of producing significantly better results than state-of-the-art unsupervised 3D reconstruction techniques. We show how 3DMiner can be applied to in-the-wild data by reconstructing shapes present in images from the LAION-5B dataset.
  arXiv  Detail & Related papers  (2023-10-29T23:08:19Z)
• Geo-SIC: Learning Deformable Geometric Shapes in Deep Image Classifiers [8.781861951759948]
  This paper presents Geo-SIC, the first deep learning model to learn deformable shapes in a deformation space for an improved performance of image classification. We introduce a newly designed framework that (i) simultaneously derives features from both image and latent shape spaces with large intra-class variations. We develop a boosted classification network, equipped with an unsupervised learning of geometric shape representations.
  arXiv  Detail & Related papers  (2022-10-25T01:55:17Z)
• Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images [82.32776379815712]
  We study the problem of shape generation in 3D mesh representation from a small number of color images with or without camera poses. We adopt to further improve the shape quality by leveraging cross-view information with a graph convolution network. Our model is robust to the quality of the initial mesh and the error of camera pose, and can be combined with a differentiable function for test-time optimization.
  arXiv  Detail & Related papers  (2022-04-21T03:42:31Z)
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
  3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry. This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
  arXiv  Detail & Related papers  (2022-04-09T03:46:18Z)
• 3D Shape Reconstruction from 2D Images with Disentangled Attribute Flow [61.62796058294777]
  Reconstructing 3D shape from a single 2D image is a challenging task. Most of the previous methods still struggle to extract semantic attributes for 3D reconstruction task. We propose 3DAttriFlow to disentangle and extract semantic attributes through different semantic levels in the input images.
  arXiv  Detail & Related papers  (2022-03-29T02:03:31Z)
• NeuralReshaper: Single-image Human-body Retouching with Deep Neural Networks [50.40798258968408]
  We present NeuralReshaper, a novel method for semantic reshaping of human bodies in single images using deep generative networks. Our approach follows a fit-then-reshape pipeline, which first fits a parametric 3D human model to a source human image. To deal with the lack-of-data problem that no paired data exist, we introduce a novel self-supervised strategy to train our network.
  arXiv  Detail & Related papers  (2022-03-20T09:02:13Z)
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction [33.95791350070165]
  Inferring 3D structure of a generic object from a 2D image is a long-standing objective of computer vision. We take an alternative approach with semi-supervised learning. That is, for a 2D image of a generic object, we decompose it into latent representations of category, shape and albedo. We show that the complete shape and albedo modeling enables us to leverage real 2D images in both modeling and model fitting.
  arXiv  Detail & Related papers  (2021-04-02T02:39:29Z)
• Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation [62.618227434286]
  We present a novel learning approach to recover the 6D poses and sizes of unseen object instances from an RGB-D image. We propose a deep network to reconstruct the 3D object model by explicitly modeling the deformation from a pre-learned categorical shape prior.
  arXiv  Detail & Related papers  (2020-07-16T16:45:05Z)
• 3D Reconstruction of Novel Object Shapes from Single Images [23.016517962380323]
  We show that our proposed SDFNet achieves state-of-the-art performance on seen and unseen shapes. We provide the first large-scale evaluation of single image shape reconstruction to unseen objects.
  arXiv  Detail & Related papers  (2020-06-14T00:34:26Z)
• Self-Supervised 2D Image to 3D Shape Translation with Disentangled Representations [92.89846887298852]
  We present a framework to translate between 2D image views and 3D object shapes. We propose SIST, a Self-supervised Image to Shape Translation framework.
  arXiv  Detail & Related papers  (2020-03-22T22:44:02Z)
• STD-Net: Structure-preserving and Topology-adaptive Deformation Network for 3D Reconstruction from a Single Image [27.885717341244014]
  3D reconstruction from a single view image is a long-standing prob-lem in computer vision. In this paper, we propose a novel methodcalled STD-Net to reconstruct the 3D models utilizing the mesh representation. Experimental results on the images from ShapeNet show that ourproposed STD-Net has better performance than other state-of-the-art methods onreconstructing 3D objects.
  arXiv  Detail & Related papers  (2020-03-07T11:02:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.