CoIn: Comprehensive 2D-3D Inpainting with Gaussian Splatting Guidance

• URL: http://arxiv.org/abs/2606.27584v1
• Date: Thu, 25 Jun 2026 22:24:27 GMT
• Title: CoIn: Comprehensive 2D-3D Inpainting with Gaussian Splatting Guidance
• Abstract summary: CoIn is a framework that bridges 2D inpainting models and 3DGS through a multi-stage consistency pipeline.<n>We introduce Reference Adaptive GS with Feature Attention to reconstruct a coarse 3D scene by adaptively weighing towards a reference view (2D -> 3D)<n>Finally, a Texture-Enhancing Discriminator refines the 3D scene to achieve high photometric realism (2D -> 3D)
• Score: 33.42204651464501
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D scene inpainting is essential for reconstructing areas corrupted by occlusions or limited viewpoints. While recent methods leverage Gaussian Splatting (GS) for efficient 3D editing, they often depend on precise multi-view segmentation masks and are inherently constrained to object removal tasks. We propose CoIn, a novel framework that bridges 2D inpainting models and 3DGS through a multi-stage consistency pipeline. Our approach first generates initial inpainted images using a diffusion model, enabling the use of arbitrary-shaped masks and diverse tasks like object insertion. We then introduce Reference Adaptive GS with Feature Attention to reconstruct a coarse 3D scene by adaptively weighing towards a reference view (2D -> 3D). This 3D representation provides geometric guidance to the diffusion process via GS-based Reference Feature Warping, ensuring multi-view consistency (3D -> 2D). Finally, a Texture-Enhancing Discriminator refines the 3D scene to achieve high photometric realism (2D -> 3D). Experiments show that CoIn, effectively leveraging bidirectional information flow, achieves state-of-the-art performance and effectively handles both object removal and object insertion with flexible mask input.

Related papers

• Benchmarking Single-Step Inpainting Methods for Multi-Object 3D Gaussian Splatting Scenes [43.95837214164744]
  Reconstruction-based inpainters outperform generative diffusion models in 3D consistency.<n>Using a state-of-the-art generative 2D inpainter, we create a straightforward baseline to underline the importance of object removal before inpainting in the 3D setting.
  arXiv  Detail & Related papers  (2026-05-29T08:24:16Z)
• Robust Prior-Guided Segmentation for Editable 3D Gaussian Splatting [6.290910602456353]
  Existing approaches that lift 2D segmentations to the 3D domain suffer from view inconsistencies and coarse masks.<n>We propose a novel framework that leverages the Segment Anything Model High Quality to generate accurate 2D masks.<n>Our approach achieves state-of-the-art segmentation accuracy and enables interactive, real-time object editing while maintaining high visual fidelity.
  arXiv  Detail & Related papers  (2026-05-15T15:29:30Z)
• A Controllable 3D Deepfake Generation Framework with Gaussian Splatting [6.969908558294805]
  We propose a novel 3D deepfake generation framework based on 3D Gaussian Splatting.<n>It enables realistic, identity-preserving face swapping and reenactment in a fully controllable 3D space.<n>Our approach bridges the gap between 3D modeling and deepfake synthesis, enabling new directions for scene-aware, controllable, and immersive visual forgeries.
  arXiv  Detail & Related papers  (2025-09-15T06:34:17Z)
• DiGA3D: Coarse-to-Fine Diffusional Propagation of Geometry and Appearance for Versatile 3D Inpainting [10.515239541326737]
  Single reference inpainting methods lack robustness when dealing with views far from the reference view.<n>Appearance inconsistency arises when independently inpainting multi-view images with 2D diffusion priors.<n>DiGA3D uses diffusion models to propagate consistent appearance and geometry in a coarse-to-fine manner.
  arXiv  Detail & Related papers  (2025-07-01T04:57:08Z)
• Visibility-Uncertainty-guided 3D Gaussian Inpainting via Scene Conceptional Learning [63.94919846010485]
  3D Gaussian inpainting (3DGI) is challenging in effectively leveraging complementary visual and semantic cues from multiple input views.<n>We propose a method that measures the visibility uncertainties of 3D points across different input views and uses them to guide 3DGI.<n>We build a novel 3DGI framework, VISTA, by integrating VISibility-uncerTainty-guided 3DGI with scene conceptuAl learning.
  arXiv  Detail & Related papers  (2025-04-23T06:21:11Z)
• Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior [57.986512832738704]
  We present a new framework Sculpt3D that equips the current pipeline with explicit injection of 3D priors from retrieved reference objects without re-training the 2D diffusion model. Specifically, we demonstrate that high-quality and diverse 3D geometry can be guaranteed by keypoints supervision through a sparse ray sampling approach. These two decoupled designs effectively harness 3D information from reference objects to generate 3D objects while preserving the generation quality of the 2D diffusion model.
  arXiv  Detail & Related papers  (2024-03-14T07:39:59Z)
• Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors [104.79392615848109]
  We present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes from a single unposed image. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture.
  arXiv  Detail & Related papers  (2023-06-30T17:59:08Z)
• Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training [65.75399500494343]
  Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for 2D and 3D computer vision. We propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training.
  arXiv  Detail & Related papers  (2023-02-27T17:56:18Z)
• SSR-2D: Semantic 3D Scene Reconstruction from 2D Images [54.46126685716471]
  In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations. The key idea of our approach is to design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images. Our method achieves the state-of-the-art performance of semantic scene completion on two large-scale benchmark datasets MatterPort3D and ScanNet.
  arXiv  Detail & Related papers  (2023-02-07T17:47:52Z)
• Cross-Modal 3D Shape Generation and Manipulation [62.50628361920725]
  We propose a generic multi-modal generative model that couples the 2D modalities and implicit 3D representations through shared latent spaces. We evaluate our framework on two representative 2D modalities of grayscale line sketches and rendered color images.
  arXiv  Detail & Related papers  (2022-07-24T19:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.