NeRF Inpainting with Geometric Diffusion Prior and Balanced Score Distillation

• URL: http://arxiv.org/abs/2411.15551v1
• Date: Sat, 23 Nov 2024 13:17:00 GMT
• Title: NeRF Inpainting with Geometric Diffusion Prior and Balanced Score Distillation
• Authors: Menglin Zhang, Xin Luo, Yunwei Lan, Chang Liu, Rui Li, Kaidong Zhang, Ganlin Yang, Dong Liu,
• Abstract summary: We introduce GB-NeRF, a novel framework that enhances NeRF inpainting through improved utilization of 2D diffusion priors. Our approach incorporates two key innovations: a fine-tuning strategy that simultaneously learns appearance and geometric priors. Our method provides superior appearance fidelity and geometric consistency compared to existing approaches.
• Score: 12.249522014019144
• License:
• Abstract: Recent advances in NeRF inpainting have leveraged pretrained diffusion models to enhance performance. However, these methods often yield suboptimal results due to their ineffective utilization of 2D diffusion priors. The limitations manifest in two critical aspects: the inadequate capture of geometric information by pretrained diffusion models and the suboptimal guidance provided by existing Score Distillation Sampling (SDS) methods. To address these problems, we introduce GB-NeRF, a novel framework that enhances NeRF inpainting through improved utilization of 2D diffusion priors. Our approach incorporates two key innovations: a fine-tuning strategy that simultaneously learns appearance and geometric priors and a specialized normal distillation loss that integrates these geometric priors into NeRF inpainting. We propose a technique called Balanced Score Distillation (BSD) that surpasses existing methods such as Score Distillation (SDS) and the improved version, Conditional Score Distillation (CSD). BSD offers improved inpainting quality in appearance and geometric aspects. Extensive experiments show that our method provides superior appearance fidelity and geometric consistency compared to existing approaches.

Related papers

• MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior [65.05773512126089]
  NeRF inpainting methods built upon explicit RGB and depth 2D inpainting supervisions are inherently constrained by the capabilities of their underlying 2D inpainters. We propose MVIP-NeRF that harnesses the potential of diffusion priors for NeRF inpainting, addressing both appearance and geometry aspects. Our experimental results show better appearance and geometry recovery than previous NeRF inpainting methods.
  arXiv  Detail & Related papers  (2024-05-05T09:04:42Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• Learn to Optimize Denoising Scores for 3D Generation: A Unified and Improved Diffusion Prior on NeRF and 3D Gaussian Splatting [60.393072253444934]
  We propose a unified framework aimed at enhancing the diffusion priors for 3D generation tasks. We identify a divergence between the diffusion priors and the training procedures of diffusion models that substantially impairs the quality of 3D generation.
  arXiv  Detail & Related papers  (2023-12-08T03:55:34Z)
• NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion [56.98287481620215]
  We present a novel method for 3D surface reconstruction from multiple images where only a part of the object of interest is captured. Our approach builds on two recent developments: surface reconstruction using neural radiance fields for the reconstruction of the visible parts of the surface, and guidance of pre-trained 2D diffusion models in the form of Score Distillation Sampling (SDS) to complete the shape in unobserved regions in a plausible manner.
  arXiv  Detail & Related papers  (2023-12-07T19:30:55Z)
• StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D [88.66678730537777]
  We present StableDreamer, a methodology incorporating three advances. First, we formalize the equivalence of the SDS generative prior and a simple supervised L2 reconstruction loss. Second, our analysis shows that while image-space diffusion contributes to geometric precision, latent-space diffusion is crucial for vivid color rendition.
  arXiv  Detail & Related papers  (2023-12-02T02:27:58Z)
• Learning A Coarse-to-Fine Diffusion Transformer for Image Restoration [39.071637725773314]
  We propose a coarse-to-fine diffusion Transformer (C2F-DFT) for image restoration. C2F-DFT contains diffusion self-attention (DFSA) and diffusion feed-forward network (DFN) In the coarse training stage, our C2F-DFT estimates noises and then generates the final clean image by a sampling algorithm.
  arXiv  Detail & Related papers  (2023-08-17T01:59:59Z)
• Diffusion Sampling with Momentum for Mitigating Divergence Artifacts [10.181486597424486]
  We investigate the potential causes of divergence artifacts and suggest that the small stability regions of numerical methods could be the principal cause. The first technique involves the incorporation of Heavy Ball (HB) momentum, a well-known technique for improving optimization, into existing diffusion numerical methods to expand their stability regions. The second technique, called Generalized Heavy Ball (GHVB), constructs a new high-order method that offers a variable trade-off between accuracy and artifact suppression.
  arXiv  Detail & Related papers  (2023-07-20T14:37:30Z)
• Low-Light Image Enhancement with Wavelet-based Diffusion Models [50.632343822790006]
  Diffusion models have achieved promising results in image restoration tasks, yet suffer from time-consuming, excessive computational resource consumption, and unstable restoration. We propose a robust and efficient Diffusion-based Low-Light image enhancement approach, dubbed DiffLL.
  arXiv  Detail & Related papers  (2023-06-01T03:08:28Z)
• DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models [23.70476220346754]
  We propose a novel guidance approach for diffusion models that uses estimated depth information derived from the rich intermediate representations of diffusion models. Experiments and extensive ablation studies demonstrate the effectiveness of our method in guiding the diffusion models toward geometrically plausible image generation.
  arXiv  Detail & Related papers  (2022-12-17T12:47:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.