Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects

• URL: http://arxiv.org/abs/2412.00518v1
• Date: Sat, 30 Nov 2024 15:58:33 GMT
• Title: Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects
• Authors: Amir Barda, Matheus Gadelha, Vladimir G. Kim, Noam Aigerman, Amit H. Bermano, Thibault Groueix,
• Abstract summary: We propose a generative technique to edit 3D shapes, represented as meshes, NeRFs, or Gaussian Splats, in approximately 3 seconds.<n>Our approach takes 3D generative editing from hours to seconds and produces higher-quality results compared to previous works.
• Score: 34.021032306348324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a generative technique to edit 3D shapes, represented as meshes, NeRFs, or Gaussian Splats, in approximately 3 seconds, without the need for running an SDS type of optimization. Our key insight is to cast 3D editing as a multiview image inpainting problem, as this representation is generic and can be mapped back to any 3D representation using the bank of available Large Reconstruction Models. We explore different fine-tuning strategies to obtain both multiview generation and inpainting capabilities within the same diffusion model. In particular, the design of the inpainting mask is an important factor of training an inpainting model, and we propose several masking strategies to mimic the types of edits a user would perform on a 3D shape. Our approach takes 3D generative editing from hours to seconds and produces higher-quality results compared to previous works.

Related papers

• InstaInpaint: Instant 3D-Scene Inpainting with Masked Large Reconstruction Model [46.67494008720215]
  InstaInpaint is a framework that produces 3D-scene inpainting from a 2D inpainting proposal within 0.4 seconds.<n>We analyze and identify several key designs that improve generalization, textural consistency, and geometric correctness.<n>InstaInpaint achieves a 1000x speed-up from prior methods while maintaining a state-of-the-art performance across two standard benchmarks.
  arXiv  Detail & Related papers  (2025-06-12T17:59:55Z)
• Visibility-Uncertainty-guided 3D Gaussian Inpainting via Scene Conceptional Learning [63.94919846010485]
  3D Gaussian inpainting (3DGI) is challenging in effectively leveraging complementary visual and semantic cues from multiple input views. We propose a method that measures the visibility uncertainties of 3D points across different input views and uses them to guide 3DGI. We build a novel 3DGI framework, VISTA, by integrating VISibility-uncerTainty-guided 3DGI with scene conceptuAl learning.
  arXiv  Detail & Related papers  (2025-04-23T06:21:11Z)
• Bolt3D: Generating 3D Scenes in Seconds [77.592919825037]
  Given one or more images, our model Bolt3D directly samples a 3D scene representation in less than seven seconds on a single GPU. Compared to prior multiview generative models that require per-scene optimization for 3D reconstruction, Bolt3D reduces the inference cost by a factor of up to 300 times.
  arXiv  Detail & Related papers  (2025-03-18T17:24:19Z)
• Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation [15.374873279207623]
  We introduce Kiss3DGen (Keep It Simple and Straightforward in 3D Generation), an efficient framework for generating, editing, and enhancing 3D objects. Specifically, we fine-tune a diffusion model to generate ''3D Bundle Image'', a tiled representation composed of multi-view images and their corresponding normal maps. This simple method effectively transforms the 3D generation problem into a 2D image generation task, maximizing the utilization of knowledge in pretrained diffusion models.
  arXiv  Detail & Related papers  (2025-03-03T10:07:19Z)
• Any-to-3D Generation via Hybrid Diffusion Supervision [67.54197818071464]
  XBind is a unified framework for any-to-3D generation using cross-modal pre-alignment techniques. XBind integrates an multimodal-aligned encoder with pre-trained diffusion models to generate 3D objects from any modalities.
  arXiv  Detail & Related papers  (2024-11-22T03:52:37Z)
• GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editing [38.948892064761914]
  GaussCtrl is a text-driven method to edit a 3D scene reconstructed by the 3D Gaussian Splatting (3DGS) Our key contribution is multi-view consistent editing, which enables editing all images together instead of iteratively editing one image.
  arXiv  Detail & Related papers  (2024-03-13T17:35:28Z)
• Real-time 3D-aware Portrait Editing from a Single Image [111.27169315556444]
  3DPE can edit a face image following given prompts, like reference images or text descriptions. A lightweight module is distilled from a 3D portrait generator and a text-to-image model.
  arXiv  Detail & Related papers  (2024-02-21T18:36:26Z)
• LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation [51.19871052619077]
  We introduce Large Multi-View Gaussian Model (LGM), a novel framework designed to generate high-resolution 3D models from text prompts or single-view images. We maintain the fast speed to generate 3D objects within 5 seconds while boosting the training resolution to 512, thereby achieving high-resolution 3D content generation.
  arXiv  Detail & Related papers  (2024-02-07T17:57:03Z)
• Efficient-NeRF2NeRF: Streamlining Text-Driven 3D Editing with Multiview Correspondence-Enhanced Diffusion Models [83.97844535389073]
  A major obstacle hindering the widespread adoption of 3D content editing is its time-intensive processing. We propose that by incorporating correspondence regularization into diffusion models, the process of 3D editing can be significantly accelerated. In most scenarios, our proposed technique brings a 10$times$ speed-up compared to the baseline method and completes the editing of a 3D scene in 2 minutes with comparable quality.
  arXiv  Detail & Related papers  (2023-12-13T23:27:17Z)
• Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion [18.67196713834323]
  This paper presents a novel approach to inpainting 3D regions of a scene, given masked multi-view images, by distilling a 2D diffusion model into a learned 3D scene representation (e.g. a NeRF) We show that this 2D diffusion model can still serve as a generative prior in a 3D multi-view reconstruction problem where we optimize a NeRF using a combination of score distillation sampling and NeRF reconstruction losses. Because our method can generate content to fill any 3D masked region, we additionally demonstrate 3D object completion, 3D object replacement, and 3D scene completion
  arXiv  Detail & Related papers  (2023-12-06T19:30:04Z)
• SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields [26.296017756560467]
  In 3D, solutions must be consistent across multiple views and geometrically valid. We propose a novel 3D inpainting method that addresses these challenges. We first demonstrate the superiority of our approach on multiview segmentation, comparing to NeRFbased methods and 2D segmentation approaches.
  arXiv  Detail & Related papers  (2022-11-22T13:14:50Z)
• RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation [68.06991943974195]
  We present RenderDiffusion, the first diffusion model for 3D generation and inference, trained using only monocular 2D supervision. We evaluate RenderDiffusion on FFHQ, AFHQ, ShapeNet and CLEVR datasets, showing competitive performance for generation of 3D scenes and inference of 3D scenes from 2D images.
  arXiv  Detail & Related papers  (2022-11-17T20:17:04Z)
• Cross-Modal 3D Shape Generation and Manipulation [62.50628361920725]
  We propose a generic multi-modal generative model that couples the 2D modalities and implicit 3D representations through shared latent spaces. We evaluate our framework on two representative 2D modalities of grayscale line sketches and rendered color images.
  arXiv  Detail & Related papers  (2022-07-24T19:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.