Mosaic-SDF for 3D Generative Models

• URL: http://arxiv.org/abs/2312.09222v2
• Date: Wed, 24 Apr 2024 10:34:45 GMT
• Title: Mosaic-SDF for 3D Generative Models
• Authors: Lior Yariv, Omri Puny, Natalia Neverova, Oran Gafni, Yaron Lipman,
• Abstract summary: When training a diffusion or flow models on 3D shapes a crucial design choice is the shape representation. We introduce Mosaic-SDF, a simple 3D shape representation that approximates the Signed Distance Function (SDF) of a given shape. We demonstrate the efficacy of the M-SDF representation by using it to train a 3D generative flow model.
• Score: 41.4585856558786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current diffusion or flow-based generative models for 3D shapes divide to two: distilling pre-trained 2D image diffusion models, and training directly on 3D shapes. When training a diffusion or flow models on 3D shapes a crucial design choice is the shape representation. An effective shape representation needs to adhere three design principles: it should allow an efficient conversion of large 3D datasets to the representation form; it should provide a good tradeoff of approximation power versus number of parameters; and it should have a simple tensorial form that is compatible with existing powerful neural architectures. While standard 3D shape representations such as volumetric grids and point clouds do not adhere to all these principles simultaneously, we advocate in this paper a new representation that does. We introduce Mosaic-SDF (M-SDF): a simple 3D shape representation that approximates the Signed Distance Function (SDF) of a given shape by using a set of local grids spread near the shape's boundary. The M-SDF representation is fast to compute for each shape individually making it readily parallelizable; it is parameter efficient as it only covers the space around the shape's boundary; and it has a simple matrix form, compatible with Transformer-based architectures. We demonstrate the efficacy of the M-SDF representation by using it to train a 3D generative flow model including class-conditioned generation with the 3D Warehouse dataset, and text-to-3D generation using a dataset of about 600k caption-shape pairs.

Related papers

• NeuSDFusion: A Spatial-Aware Generative Model for 3D Shape Completion, Reconstruction, and Generation [52.772319840580074]
  3D shape generation aims to produce innovative 3D content adhering to specific conditions and constraints. Existing methods often decompose 3D shapes into a sequence of localized components, treating each element in isolation. We introduce a novel spatial-aware 3D shape generation framework that leverages 2D plane representations for enhanced 3D shape modeling.
  arXiv  Detail & Related papers  (2024-03-27T04:09:34Z)
• Hybrid Neural Diffeomorphic Flow for Shape Representation and Generation via Triplane [16.684276798449115]
  HNDF is a method that implicitly learns the underlying representation and decomposes intricate dense correspondences into explicitly axis-aligned triplane features. Unlike conventional approaches that directly generate new 3D shapes, we explore the idea of shape generation with deformed template shape via diffeomorphic flows.
  arXiv  Detail & Related papers  (2023-07-04T23:28:01Z)
• Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation [47.945556996219295]
  We present a novel alignment-before-generation approach to generate 3D shapes based on 2D images or texts. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM)
  arXiv  Detail & Related papers  (2023-06-29T17:17:57Z)
• Locally Attentional SDF Diffusion for Controllable 3D Shape Generation [24.83724829092307]
  We propose a diffusion-based 3D generation framework, to model plausible 3D shapes, via 2D sketch image input. Our method is built on a two-stage diffusion model. The first stage, named occupancy-diffusion, aims to generate a low-resolution occupancy field to approximate the shape shell. The second stage, named SDF-diffusion, synthesizes a high-resolution signed distance field within the occupied voxels determined by the first stage to extract fine geometry.
  arXiv  Detail & Related papers  (2023-05-08T05:07:23Z)
• Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis [90.26556260531707]
  DMTet is a conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology.
  arXiv  Detail & Related papers  (2021-11-08T05:29:35Z)
• Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes [77.6741486264257]
  We introduce an efficient neural representation that, for the first time, enables real-time rendering of high-fidelity neural SDFs. We show that our representation is 2-3 orders of magnitude more efficient in terms of rendering speed compared to previous works.
  arXiv  Detail & Related papers  (2021-01-26T18:50:22Z)
• Deep Implicit Templates for 3D Shape Representation [70.9789507686618]
  We propose a new 3D shape representation that supports explicit correspondence reasoning in deep implicit representations. Our key idea is to formulate DIFs as conditional deformations of a template implicit function. We show that our method can not only learn a common implicit template for a collection of shapes, but also establish dense correspondences across all the shapes simultaneously without any supervision.
  arXiv  Detail & Related papers  (2020-11-30T06:01:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.