Sketch-guided Cage-based 3D Gaussian Splatting Deformation

• URL: http://arxiv.org/abs/2411.12168v2
• Date: Fri, 22 Nov 2024 15:25:13 GMT
• Title: Sketch-guided Cage-based 3D Gaussian Splatting Deformation
• Authors: Tianhao Xie, Noam Aigerman, Eugene Belilovsky, Tiberiu Popa,
• Abstract summary: 3D Gaussian Splatting (GS) is one of the most promising novel 3D representations. We present a sketch-guided 3D GS deformation system that allows users to intuitively modify the geometry of a 3D GS model.
• Score: 19.39701978836869
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D Gaussian Splatting (GS) is one of the most promising novel 3D representations that has received great interest in computer graphics and computer vision. While various systems have introduced editing capabilities for 3D GS, such as those guided by text prompts, fine-grained control over deformation remains an open challenge. In this work, we present a novel sketch-guided 3D GS deformation system that allows users to intuitively modify the geometry of a 3D GS model by drawing a silhouette sketch from a single viewpoint. Our approach introduces a new deformation method that combines cage-based deformations with a variant of Neural Jacobian Fields, enabling precise, fine-grained control. Additionally, it leverages large-scale 2D diffusion priors and ControlNet to ensure the generated deformations are semantically plausible. Through a series of experiments, we demonstrate the effectiveness of our method and showcase its ability to animate static 3D GS models as one of its key applications.

Related papers

• CAGE-GS: High-fidelity Cage Based 3D Gaussian Splatting Deformation [7.218737495375119]
  CAGE-GS is a cage-based 3DGS deformation method that seamlessly aligns a source 3DGS scene with a user-defined target shape. Our approach learns a deformation cage from the target, which guides the geometric transformation of the source scene. Our method is highly flexible, accommodating various target shape representations, including texts, images, point clouds, meshes and 3DGS models.
  arXiv  Detail & Related papers  (2025-04-17T10:00:15Z)
• RigGS: Rigging of 3D Gaussians for Modeling Articulated Objects in Videos [50.37136267234771]
  RigGS is a new paradigm that leverages 3D Gaussian representation and skeleton-based motion representation to model dynamic objects. Our method can generate realistic new actions easily for objects and achieve high-quality rendering.
  arXiv  Detail & Related papers  (2025-03-21T03:27:07Z)
• Generating Editable Head Avatars with 3D Gaussian GANs [57.51487984425395]
  Traditional 3D-aware generative adversarial networks (GANs) achieve photorealistic and view-consistent 3D head synthesis. We propose a novel approach that enhances the editability and animation control of 3D head avatars by incorporating 3D Gaussian Splatting (3DGS) as an explicit 3D representation. Our approach delivers high-quality 3D-aware synthesis with state-of-the-art controllability.
  arXiv  Detail & Related papers  (2024-12-26T10:10:03Z)
• 3D Gaussian Editing with A Single Image [19.662680524312027]
  We introduce a novel single-image-driven 3D scene editing approach based on 3D Gaussian Splatting. Our method learns to optimize the 3D Gaussians to align with an edited version of the image rendered from a user-specified viewpoint. Experiments show the effectiveness of our method in handling geometric details, long-range, and non-rigid deformation.
  arXiv  Detail & Related papers  (2024-08-14T13:17:42Z)
• GSD: View-Guided Gaussian Splatting Diffusion for 3D Reconstruction [52.04103235260539]
  We present a diffusion model approach based on Gaussian Splatting representation for 3D object reconstruction from a single view. The model learns to generate 3D objects represented by sets of GS ellipsoids. The final reconstructed objects explicitly come with high-quality 3D structure and texture, and can be efficiently rendered in arbitrary views.
  arXiv  Detail & Related papers  (2024-07-05T03:43:08Z)
• GSDeformer: Direct Cage-based Deformation for 3D Gaussian Splatting [3.8859983281943116]
  Our method achieves free-form deformation on 3D Gaussian Splatting(2DGS) without requiring any architectural changes. We convert 3DGS into a novel proxy point cloud representation, where its deformation can be used to infer transformations to apply on the 3D gaussians making up 3DGS. Our method does not modify the underlying architecture of 3DGS.
  arXiv  Detail & Related papers  (2024-05-24T12:16:28Z)
• Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning [52.81032340916171]
  Coin3D allows users to control the 3D generation using a coarse geometry proxy assembled from basic shapes. Our method achieves superior controllability and flexibility in the 3D assets generation task.
  arXiv  Detail & Related papers  (2024-05-13T17:56:13Z)
• Direct Learning of Mesh and Appearance via 3D Gaussian Splatting [3.4899193297791054]
  We propose a learnable scene model that incorporates 3DGS with an explicit geometry representation, namely a mesh. Our model learns the mesh and appearance in an end-to-end manner, where we bind 3D Gaussians to the mesh faces and perform differentiable rendering of 3DGS to obtain photometric supervision.
  arXiv  Detail & Related papers  (2024-05-11T07:56:19Z)
• 3D Geometry-aware Deformable Gaussian Splatting for Dynamic View Synthesis [49.352765055181436]
  We propose a 3D geometry-aware deformable Gaussian Splatting method for dynamic view synthesis. Our solution achieves 3D geometry-aware deformation modeling, which enables improved dynamic view synthesis and 3D dynamic reconstruction.
  arXiv  Detail & Related papers  (2024-04-09T12:47:30Z)
• Mesh-based Gaussian Splatting for Real-time Large-scale Deformation [58.18290393082119]
  It is challenging for users to directly deform or manipulate implicit representations with large deformations in the real-time fashion. We develop a novel GS-based method that enables interactive deformation. Our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate.
  arXiv  Detail & Related papers  (2024-02-07T12:36:54Z)
• A Survey on 3D Gaussian Splatting [51.96747208581275]
  3D Gaussian splatting (GS) has emerged as a transformative technique in the realm of explicit radiance field and computer graphics. We provide the first systematic overview of the recent developments and critical contributions in the domain of 3D GS. By enabling unprecedented rendering speed, 3D GS opens up a plethora of applications, ranging from virtual reality to interactive media and beyond.
  arXiv  Detail & Related papers  (2024-01-08T13:42:59Z)
• Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars [36.4402388864691]
  3D-aware generative adversarial networks (GANs) synthesize high-fidelity and multi-view-consistent facial images using only collections of single-view 2D imagery. Recent efforts incorporate 3D Morphable Face Model (3DMM) to describe deformation in generative radiance fields either explicitly or implicitly. We propose a novel 3D GAN framework for unsupervised learning of generative, high-quality and 3D-consistent facial avatars from unstructured 2D images.
  arXiv  Detail & Related papers  (2022-11-21T06:40:46Z)
• Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian [58.704089101826774]
  We present a 3D-aware image deformation method with minimal restrictions on shape category and deformation type. We take a supervised learning-based approach to predict the shape Laplacian of the underlying volume of a 3D reconstruction represented as a point cloud. In the experiments, we present our results of deforming 2D character and clothed human images.
  arXiv  Detail & Related papers  (2022-03-29T04:57:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.