PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models
- URL: http://arxiv.org/abs/2312.11417v1
- Date: Mon, 18 Dec 2023 18:19:26 GMT
- Title: PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models
- Authors: Antonio Alliegro, Yawar Siddiqui, Tatiana Tommasi, Matthias
Nie{\ss}ner
- Abstract summary: PolyDiff is the first diffusion-based approach capable of directly generating realistic and diverse 3D polygonal meshes.
Our model is capable of producing high-quality 3D polygonal meshes, ready for integration into downstream 3D.
- Score: 15.846449180313778
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce PolyDiff, the first diffusion-based approach capable of directly
generating realistic and diverse 3D polygonal meshes. In contrast to methods
that use alternate 3D shape representations (e.g. implicit representations),
our approach is a discrete denoising diffusion probabilistic model that
operates natively on the polygonal mesh data structure. This enables learning
of both the geometric properties of vertices and the topological
characteristics of faces. Specifically, we treat meshes as quantized triangle
soups, progressively corrupted with categorical noise in the forward diffusion
phase. In the reverse diffusion phase, a transformer-based denoising network is
trained to revert the noising process, restoring the original mesh structure.
At inference, new meshes can be generated by applying this denoising network
iteratively, starting with a completely noisy triangle soup. Consequently, our
model is capable of producing high-quality 3D polygonal meshes, ready for
integration into downstream 3D workflows. Our extensive experimental analysis
shows that PolyDiff achieves a significant advantage (avg. FID and JSD
improvement of 18.2 and 5.8 respectively) over current state-of-the-art
methods.
Related papers
- MeshXL: Neural Coordinate Field for Generative 3D Foundation Models [51.1972329762843]
We present a family of generative pre-trained auto-regressive models, which addresses the process of 3D mesh generation with modern large language model approaches.
MeshXL is able to generate high-quality 3D meshes, and can also serve as foundation models for various down-stream applications.
arXiv Detail & Related papers (2024-05-31T14:35:35Z) - Hybrid Neural Diffeomorphic Flow for Shape Representation and Generation
via Triplane [16.684276798449115]
HNDF is a method that implicitly learns the underlying representation and decomposes intricate dense correspondences into explicitly axis-aligned triplane features.
Unlike conventional approaches that directly generate new 3D shapes, we explore the idea of shape generation with deformed template shape via diffeomorphic flows.
arXiv Detail & Related papers (2023-07-04T23:28:01Z) - MeshDiffusion: Score-based Generative 3D Mesh Modeling [68.40770889259143]
We consider the task of generating realistic 3D shapes for automatic scene generation and physical simulation.
We take advantage of the graph structure of meshes and use a simple yet very effective generative modeling method to generate 3D meshes.
Specifically, we represent meshes with deformable tetrahedral grids, and then train a diffusion model on this direct parametrization.
arXiv Detail & Related papers (2023-03-14T17:59:01Z) - Neural Wavelet-domain Diffusion for 3D Shape Generation, Inversion, and
Manipulation [54.09274684734721]
We present a new approach for 3D shape generation, inversion, and manipulation, through a direct generative modeling on a continuous implicit representation in wavelet domain.
Specifically, we propose a compact wavelet representation with a pair of coarse and detail coefficient volumes to implicitly represent 3D shapes via truncated signed distance functions and multi-scale biorthogonal wavelets.
We may jointly train an encoder network to learn a latent space for inverting shapes, allowing us to enable a rich variety of whole-shape and region-aware shape manipulations.
arXiv Detail & Related papers (2023-02-01T02:47:53Z) - 3D Neural Field Generation using Triplane Diffusion [37.46688195622667]
We present an efficient diffusion-based model for 3D-aware generation of neural fields.
Our approach pre-processes training data, such as ShapeNet meshes, by converting them to continuous occupancy fields.
We demonstrate state-of-the-art results on 3D generation on several object classes from ShapeNet.
arXiv Detail & Related papers (2022-11-30T01:55:52Z) - Neural Wavelet-domain Diffusion for 3D Shape Generation [52.038346313823524]
This paper presents a new approach for 3D shape generation, enabling direct generative modeling on a continuous implicit representation in wavelet domain.
Specifically, we propose a compact wavelet representation with a pair of coarse and detail coefficient volumes to implicitly represent 3D shapes via truncated signed distance functions and multi-scale biorthogonal wavelets.
arXiv Detail & Related papers (2022-09-19T02:51:48Z) - Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D
Shape Synthesis [90.26556260531707]
DMTet is a conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels.
Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology.
arXiv Detail & Related papers (2021-11-08T05:29:35Z) - 3D Shape Generation and Completion through Point-Voxel Diffusion [24.824065748889048]
We propose a novel approach for probabilistic generative modeling of 3D shapes.
Point-Voxel Diffusion (PVD) is a unified, probabilistic formulation for unconditional shape generation and conditional, multimodal shape completion.
PVD can be viewed as a series of denoising steps, reversing the diffusion process from observed point cloud data to Gaussian noise, and is trained by optimizing a variational lower bound to the (conditional) likelihood function.
arXiv Detail & Related papers (2021-04-08T10:38:03Z) - Dense Non-Rigid Structure from Motion: A Manifold Viewpoint [162.88686222340962]
Non-Rigid Structure-from-Motion (NRSfM) problem aims to recover 3D geometry of a deforming object from its 2D feature correspondences across multiple frames.
We show that our approach significantly improves accuracy, scalability, and robustness against noise.
arXiv Detail & Related papers (2020-06-15T09:15:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.