3DILG: Irregular Latent Grids for 3D Generative Modeling
- URL: http://arxiv.org/abs/2205.13914v1
- Date: Fri, 27 May 2022 11:29:52 GMT
- Title: 3DILG: Irregular Latent Grids for 3D Generative Modeling
- Authors: Biao Zhang, Matthias Nie{\ss}ner, Peter Wonka
- Abstract summary: We propose a new representation for encoding 3D shapes as neural fields.
The representation is designed to be compatible with the transformer architecture and to benefit both shape reconstruction and shape generation.
- Score: 44.16807313707137
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose a new representation for encoding 3D shapes as neural fields. The
representation is designed to be compatible with the transformer architecture
and to benefit both shape reconstruction and shape generation. Existing works
on neural fields are grid-based representations with latents defined on a
regular grid. In contrast, we define latents on irregular grids, enabling our
representation to be sparse and adaptive. In the context of shape
reconstruction from point clouds, our shape representation built on irregular
grids improves upon grid-based methods in terms of reconstruction accuracy. For
shape generation, our representation promotes high-quality shape generation
using auto-regressive probabilistic models. We show different applications that
improve over the current state of the art. First, we show results for
probabilistic shape reconstruction from a single higher resolution image.
Second, we train a probabilistic model conditioned on very low resolution
images. Third, we apply our model to category-conditioned generation. All
probabilistic experiments confirm that we are able to generate detailed and
high quality shapes to yield the new state of the art in generative 3D shape
modeling.
Related papers
- Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability [118.26563926533517]
Auto-regressive models have achieved impressive results in 2D image generation by modeling joint distributions in grid space.
We extend auto-regressive models to 3D domains, and seek a stronger ability of 3D shape generation by improving auto-regressive models at capacity and scalability simultaneously.
arXiv Detail & Related papers (2024-02-19T15:33:09Z) - Make-A-Shape: a Ten-Million-scale 3D Shape Model [52.701745578415796]
This paper introduces Make-A-Shape, a new 3D generative model designed for efficient training on a vast scale.
We first innovate a wavelet-tree representation to compactly encode shapes by formulating the subband coefficient filtering scheme.
We derive the subband adaptive training strategy to train our model to effectively learn to generate coarse and detail wavelet coefficients.
arXiv Detail & Related papers (2024-01-20T00:21:58Z) - Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text
Aligned Latent Representation [47.945556996219295]
We present a novel alignment-before-generation approach to generate 3D shapes based on 2D images or texts.
Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM)
arXiv Detail & Related papers (2023-06-29T17:17:57Z) - Learning Versatile 3D Shape Generation with Improved AR Models [91.87115744375052]
Auto-regressive (AR) models have achieved impressive results in 2D image generation by modeling joint distributions in the grid space.
We propose the Improved Auto-regressive Model (ImAM) for 3D shape generation, which applies discrete representation learning based on a latent vector instead of volumetric grids.
arXiv Detail & Related papers (2023-03-26T12:03:18Z) - 3DShape2VecSet: A 3D Shape Representation for Neural Fields and
Generative Diffusion Models [42.928400751670935]
We introduce 3DShape2VecSet, a novel shape representation for neural fields designed for generative diffusion models.
Our results show improved performance in 3D shape encoding and 3D shape generative modeling tasks.
arXiv Detail & Related papers (2023-01-26T22:23:03Z) - 3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models [8.583859530633417]
We propose a diffusion model for neural implicit representations of 3D shapes that operates in the latent space of an auto-decoder.
This allows us to generate diverse and high quality 3D surfaces.
arXiv Detail & Related papers (2022-12-01T20:00:00Z) - Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images [82.32776379815712]
We study the problem of shape generation in 3D mesh representation from a small number of color images with or without camera poses.
We adopt to further improve the shape quality by leveraging cross-view information with a graph convolution network.
Our model is robust to the quality of the initial mesh and the error of camera pose, and can be combined with a differentiable function for test-time optimization.
arXiv Detail & Related papers (2022-04-21T03:42:31Z) - Autoregressive 3D Shape Generation via Canonical Mapping [92.91282602339398]
transformers have shown remarkable performances in a variety of generative tasks such as image, audio, and text generation.
In this paper, we aim to further exploit the power of transformers and employ them for the task of 3D point cloud generation.
Our model can be easily extended to multi-modal shape completion as an application for conditional shape generation.
arXiv Detail & Related papers (2022-04-05T03:12:29Z) - PolyGen: An Autoregressive Generative Model of 3D Meshes [22.860421649320287]
We present an approach which models the mesh directly using a Transformer-based architecture.
Our model can condition on a range of inputs, including object classes, voxels, and images.
We show that the model is capable of producing high-quality, usable meshes, and establish log-likelihood benchmarks for the mesh-modelling task.
arXiv Detail & Related papers (2020-02-23T17:16:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.