Point'n Move: Interactive Scene Object Manipulation on Gaussian
Splatting Radiance Fields
- URL: http://arxiv.org/abs/2311.16737v1
- Date: Tue, 28 Nov 2023 12:33:49 GMT
- Title: Point'n Move: Interactive Scene Object Manipulation on Gaussian
Splatting Radiance Fields
- Authors: Jiajun Huang, Hongchuan Yu
- Abstract summary: Point'n Move is a method that achieves interactive scene object manipulation with exposed region inpainting.
We adopt Gaussian Splatting Radiance Field as the scene representation and fully leverage its explicit nature and speed advantage.
- Score: 4.5907922403638945
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose Point'n Move, a method that achieves interactive scene object
manipulation with exposed region inpainting. Interactivity here further comes
from intuitive object selection and real-time editing. To achieve this, we
adopt Gaussian Splatting Radiance Field as the scene representation and fully
leverage its explicit nature and speed advantage. Its explicit representation
formulation allows us to devise a 2D prompt points to 3D mask dual-stage
self-prompting segmentation algorithm, perform mask refinement and merging,
minimize change as well as provide good initialization for scene inpainting and
perform editing in real-time without per-editing training, all leads to
superior quality and performance. We test our method by performing editing on
both forward-facing and 360 scenes. We also compare our method against existing
scene object removal methods, showing superior quality despite being more
capable and having a speed advantage.
Related papers
- 3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting [100.94916668527544]
Existing methods solely focus on either 2D individual object or 3D global scene editing.
We propose 3DitScene, a novel and unified scene editing framework.
It enables seamless editing from 2D to 3D, allowing precise control over scene composition and individual objects.
arXiv Detail & Related papers (2024-05-28T17:59:01Z) - RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting [63.567363455092234]
RefFusion is a novel 3D inpainting method based on a multi-scale personalization of an image inpainting diffusion model to the given reference view.
Our framework achieves state-of-the-art results for object removal while maintaining high controllability.
arXiv Detail & Related papers (2024-04-16T17:50:02Z) - ZONE: Zero-Shot Instruction-Guided Local Editing [56.56213730578504]
We propose a Zero-shot instructiON-guided local image Editing approach, termed ZONE.
We first convert the editing intent from the user-provided instruction into specific image editing regions through InstructPix2Pix.
We then propose a Region-IoU scheme for precise image layer extraction from an off-the-shelf segment model.
arXiv Detail & Related papers (2023-12-28T02:54:34Z) - LatentEditor: Text Driven Local Editing of 3D Scenes [8.966537479017951]
We introduce textscLatentEditor, a framework for precise and locally controlled editing of neural fields using text prompts.
We successfully embed real-world scenes into the latent space, resulting in a faster and more adaptable NeRF backbone for editing.
Our approach achieves faster editing speeds and superior output quality compared to existing 3D editing models.
arXiv Detail & Related papers (2023-12-14T19:38:06Z) - Neural Implicit Field Editing Considering Object-environment Interaction [5.285267388811263]
We propose an Object and Scene environment Interaction aware (OSI-aware) system.
It is a novel two-stream neural rendering system considering object and scene environment interaction.
It achieves competitive performance for the rendering quality in novel-view synthesis tasks.
arXiv Detail & Related papers (2023-11-01T10:35:47Z) - OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation
with Neural Radiance Fields [53.32527220134249]
The emergence of Neural Radiance Fields (NeRF) for novel view synthesis has increased interest in 3D scene editing.
Current methods face challenges such as time-consuming object labeling, limited capability to remove specific targets, and compromised rendering quality after removal.
This paper proposes a novel object-removing pipeline, named OR-NeRF, that can remove objects from 3D scenes with user-given points or text prompts on a single view.
arXiv Detail & Related papers (2023-05-17T18:18:05Z) - SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural
Radiance Fields [26.296017756560467]
In 3D, solutions must be consistent across multiple views and geometrically valid.
We propose a novel 3D inpainting method that addresses these challenges.
We first demonstrate the superiority of our approach on multiview segmentation, comparing to NeRFbased methods and 2D segmentation approaches.
arXiv Detail & Related papers (2022-11-22T13:14:50Z) - Video2StyleGAN: Encoding Video in Latent Space for Manipulation [63.03250800510085]
We propose a novel network to encode face videos into the latent space of StyleGAN for semantic face video manipulation.
Our approach can significantly outperform existing single image methods, while achieving real-time (66 fps) speed.
arXiv Detail & Related papers (2022-06-27T06:48:15Z) - Pixel Sampling for Style Preserving Face Pose Editing [53.14006941396712]
We present a novel two-stage approach to solve the dilemma, where the task of face pose manipulation is cast into face inpainting.
By selectively sampling pixels from the input face and slightly adjust their relative locations, the face editing result faithfully keeps the identity information as well as the image style unchanged.
With the 3D facial landmarks as guidance, our method is able to manipulate face pose in three degrees of freedom, i.e., yaw, pitch, and roll, resulting in more flexible face pose editing.
arXiv Detail & Related papers (2021-06-14T11:29:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.