MicroDreamer: Zero-shot 3D Generation in $\sim$20 Seconds by Score-based Iterative Reconstruction

• URL: http://arxiv.org/abs/2404.19525v2
• Date: Tue, 28 May 2024 01:42:13 GMT
• Title: MicroDreamer: Zero-shot 3D Generation in $\sim$20 Seconds by Score-based Iterative Reconstruction
• Authors: Luxi Chen, Zhengyi Wang, Zihan Zhou, Tingting Gao, Hang Su, Jun Zhu, Chongxuan Li,
• Abstract summary: We introduce score-based iterative reconstruction (SIR), an efficient and general algorithm mimicking a differentiable 3D reconstruction process to reduce the NFEs. We present an efficient approach called MicroDreamer that generally applies to various 3D representations and 3D generation tasks.
• Score: 37.07128043394227
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimization-based approaches, such as score distillation sampling (SDS), show promise in zero-shot 3D generation but suffer from low efficiency, primarily due to the high number of function evaluations (NFEs) required for each sample. In this paper, we introduce score-based iterative reconstruction (SIR), an efficient and general algorithm mimicking a differentiable 3D reconstruction process to reduce the NFEs. Given a single set of images sampled from a multi-view score-based diffusion model, SIR repeatedly optimizes 3D parameters, unlike the single-step optimization in SDS. With other improvements in training, we present an efficient approach called MicroDreamer that generally applies to various 3D representations and 3D generation tasks. In particular, retaining a comparable performance, MicroDreamer is 5-20 times faster than SDS in generating neural radiance field and takes about 20 seconds to generate meshes from 3D Gaussian splatting on a single A100 GPU, halving the time of the fastest zero-shot baseline, DreamGaussian. Our code is available at \url{https://github.com/ML-GSAI/MicroDreamer}.

Related papers

• Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstruction [153.52406455209538]
  Gamba is an end-to-end 3D reconstruction model from a single-view image. It completes reconstruction within 0.05 seconds on a single NVIDIA A100 GPU.
  arXiv  Detail & Related papers  (2024-03-27T17:40:14Z)
• AGG: Amortized Generative 3D Gaussians for Single Image to 3D [108.38567665695027]
  We introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image. AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. We propose a cascaded pipeline that first generates a coarse representation of the 3D data and later upsamples it with a 3D Gaussian super-resolution module.
  arXiv  Detail & Related papers  (2024-01-08T18:56:33Z)
• Splatter Image: Ultra-Fast Single-View 3D Reconstruction [67.96212093828179]
  Splatter Image is based on Gaussian Splatting, which allows fast and high-quality reconstruction of 3D scenes from multiple images. We learn a neural network that, at test time, performs reconstruction in a feed-forward manner, at 38 FPS. On several synthetic, real, multi-category and large-scale benchmark datasets, we achieve better results in terms of PSNR, LPIPS, and other metrics while training and evaluating much faster than prior works.
  arXiv  Detail & Related papers  (2023-12-20T16:14:58Z)
• DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling [69.84233945649194]
  We propose DreamPropeller, a drop-in acceleration algorithm that can be wrapped around any existing text-to-3D generation pipeline based on score distillation. Our framework generalizes Picard iterations, a classical algorithm for parallel sampling an ODE path. We show that our algorithm trades parallel compute for wallclock time and empirically achieves up to 4.7x speedup with a negligible drop in generation quality for all tested frameworks.
  arXiv  Detail & Related papers  (2023-11-28T01:28:58Z)
• DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation [55.661467968178066]
  We propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks.
  arXiv  Detail & Related papers  (2023-09-28T17:55:05Z)
• GPU optimization of the 3D Scale-invariant Feature Transform Algorithm and a Novel BRIEF-inspired 3D Fast Descriptor [5.1537294207900715]
  This work details a highly efficient implementation of the 3D scale-invariant feature transform (SIFT) algorithm, for the purpose of machine learning from large sets of medical image data. The primary operations of the 3D SIFT code are implemented on a graphics processing unit (GPU), including convolution, sub-sampling, and 4D peak detection from scale-space pyramids. The performance improvements are quantified in keypoint detection and image-to-image matching experiments, using 3D MRI human brain volumes of different people.
  arXiv  Detail & Related papers  (2021-12-19T20:56:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.