Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models

• URL: http://arxiv.org/abs/2401.00208v1
• Date: Sat, 30 Dec 2023 11:26:55 GMT
• Title: Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models
• Authors: Han Jiang, Haosen Sun, Ruoxuan Li, Chi-Keung Tang, Yu-Wing Tai
• Abstract summary: Current Neural Radiance Fields (NeRF) can generate photorealistic novel views. This paper proposes Inpaint4DNeRF to capitalize on state-of-the-art stable diffusion models.
• Score: 59.96172701917538
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current Neural Radiance Fields (NeRF) can generate photorealistic novel views. For editing 3D scenes represented by NeRF, with the advent of generative models, this paper proposes Inpaint4DNeRF to capitalize on state-of-the-art stable diffusion models (e.g., ControlNet) for direct generation of the underlying completed background content, regardless of static or dynamic. The key advantages of this generative approach for NeRF inpainting are twofold. First, after rough mask propagation, to complete or fill in previously occluded content, we can individually generate a small subset of completed images with plausible content, called seed images, from which simple 3D geometry proxies can be derived. Second and the remaining problem is thus 3D multiview consistency among all completed images, now guided by the seed images and their 3D proxies. Without other bells and whistles, our generative Inpaint4DNeRF baseline framework is general which can be readily extended to 4D dynamic NeRFs, where temporal consistency can be naturally handled in a similar way as our multiview consistency.

Related papers

• Denoising Diffusion via Image-Based Rendering [54.20828696348574]
  We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
  arXiv  Detail & Related papers  (2024-02-05T19:00:45Z)
• SIGNeRF: Scene Integrated Generation for Neural Radiance Fields [1.1037667460077816]
  We propose a novel approach for fast and controllable NeRF scene editing and scene-integrated object generation. A new generative update strategy ensures 3D consistency across the edited images, without requiring iterative optimization. By exploiting the depth conditioning mechanism of the image diffusion model, we gain fine control over the spatial location of the edit.
  arXiv  Detail & Related papers  (2024-01-03T09:46:43Z)
• RePaint-NeRF: NeRF Editting via Semantic Masks and Diffusion Models [36.236190350126826]
  We propose a novel framework that can take RGB images as input and alter the 3D content in neural scenes. Specifically, we semantically select the target object and a pre-trained diffusion model will guide the NeRF model to generate new 3D objects. Experiment results show that our algorithm is effective for editing 3D objects in NeRF under different text prompts.
  arXiv  Detail & Related papers  (2023-06-09T04:49:31Z)
• ZIGNeRF: Zero-shot 3D Scene Representation with Invertible Generative Neural Radiance Fields [2.458437232470188]
  We introduce ZIGNeRF, an innovative model that executes zero-shot Generative Adrial Network (GAN)versa for the generation of multi-view images from a single out-of-domain image. ZIGNeRF is capable of disentangling the object from the background and executing 3D operations such as 360-degree rotation or depth and horizontal translation.
  arXiv  Detail & Related papers  (2023-06-05T09:41:51Z)
• Registering Neural Radiance Fields as 3D Density Images [55.64859832225061]
  We propose to use universal pre-trained neural networks that can be trained and tested on different scenes. We demonstrate that our method, as a global approach, can effectively register NeRF models.
  arXiv  Detail & Related papers  (2023-05-22T09:08:46Z)
• TextMesh: Generation of Realistic 3D Meshes From Text Prompts [56.2832907275291]
  We propose a novel method for generation of highly realistic-looking 3D meshes. To this end, we extend NeRF to employ an SDF backbone, leading to improved 3D mesh extraction.
  arXiv  Detail & Related papers  (2023-04-24T20:29:41Z)
• Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction [77.69363640021503]
  3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. We present SSDNeRF, a unified approach that employs an expressive diffusion model to learn a generalizable prior of neural radiance fields (NeRF) from multi-view images of diverse objects.
  arXiv  Detail & Related papers  (2023-04-13T17:59:01Z)
• NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors [24.05480789681139]
  We propose NeRDi, a single-view NeRF synthesis framework with general image priors from 2D diffusion models. We leverage off-the-shelf vision-language models and introduce a two-section language guidance as conditioning inputs to the diffusion model. We also demonstrate our generalizability in zero-shot NeRF synthesis for in-the-wild images.
  arXiv  Detail & Related papers  (2022-12-06T19:00:07Z)
• Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures [72.44361273600207]
  We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models. Latent Diffusion Models apply the entire diffusion process in a compact latent space of a pretrained autoencoder. We show that latent score distillation can be successfully applied directly on 3D meshes.
  arXiv  Detail & Related papers  (2022-11-14T18:25:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.