ART3D: 3D Gaussian Splatting for Text-Guided Artistic Scenes Generation

• URL: http://arxiv.org/abs/2405.10508v1
• Date: Fri, 17 May 2024 03:19:36 GMT
• Title: ART3D: 3D Gaussian Splatting for Text-Guided Artistic Scenes Generation
• Authors: Pengzhi Li, Chengshuai Tang, Qinxuan Huang, Zhiheng Li,
• Abstract summary: ART3D is a novel framework that combines diffusion models and 3D Gaussian splatting techniques. By leveraging depth information and an initial artistic image, we generate a point cloud map. We also propose a depth consistency module to enhance 3D scene consistency.
• Score: 18.699440994076003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we explore the existing challenges in 3D artistic scene generation by introducing ART3D, a novel framework that combines diffusion models and 3D Gaussian splatting techniques. Our method effectively bridges the gap between artistic and realistic images through an innovative image semantic transfer algorithm. By leveraging depth information and an initial artistic image, we generate a point cloud map, addressing domain differences. Additionally, we propose a depth consistency module to enhance 3D scene consistency. Finally, the 3D scene serves as initial points for optimizing Gaussian splats. Experimental results demonstrate ART3D's superior performance in both content and structural consistency metrics when compared to existing methods. ART3D significantly advances the field of AI in art creation by providing an innovative solution for generating high-quality 3D artistic scenes.

Related papers

• AI-Driven Stylization of 3D Environments [1.1097407843705314]
  We discuss methods to stylize a scene of 3D primitive objects into a higher fidelity 3D scene using novel 3D representations like NeRFs and 3D Gaussian Splatting. Our approach leverages existing image stylization systems and image-to-3D generative models to create a pipeline that iteratively stylizes and composites 3D objects into scenes.
  arXiv  Detail & Related papers  (2024-11-09T04:51:17Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• Control3D: Towards Controllable Text-to-3D Generation [107.81136630589263]
  We present a text-to-3D generation conditioning on the additional hand-drawn sketch, namely Control3D. A 2D conditioned diffusion model (ControlNet) is remoulded to guide the learning of 3D scene parameterized as NeRF. We exploit a pre-trained differentiable photo-to-sketch model to directly estimate the sketch of the rendered image over synthetic 3D scene.
  arXiv  Detail & Related papers  (2023-11-09T15:50:32Z)
• Guide3D: Create 3D Avatars from Text and Image Guidance [55.71306021041785]
  Guide3D is a text-and-image-guided generative model for 3D avatar generation based on diffusion models. Our framework produces topologically and structurally correct geometry and high-resolution textures.
  arXiv  Detail & Related papers  (2023-08-18T17:55:47Z)
• ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections [71.46546520120162]
  Estimating 3D articulated shapes like animal bodies from monocular images is inherently challenging. We propose ARTIC3D, a self-supervised framework to reconstruct per-instance 3D shapes from a sparse image collection in-the-wild. We produce realistic animations by fine-tuning the rendered shape and texture under rigid part transformations.
  arXiv  Detail & Related papers  (2023-06-07T17:47:50Z)
• DITTO-NeRF: Diffusion-based Iterative Text To Omni-directional 3D Model [15.091263190886337]
  We propose a novel pipeline to generate a high-quality 3D NeRF model from a text prompt or a single image. DitTO-NeRF consists of constructing high-quality partial 3D object for limited in-boundary (IB) angles using the given or text-generated 2D image from the frontal view. We propose progressive 3D object reconstruction schemes in terms of scales (low to high resolution), angles (IB angles initially to outer-boundary (OB) later, and masks (object to background boundary) in our DITTO-NeRF.
  arXiv  Detail & Related papers  (2023-04-06T02:27:22Z)
• Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior [36.40582157854088]
  In this work, we investigate the problem of creating high-fidelity 3D content from only a single image. We leverage prior knowledge from a well-trained 2D diffusion model to act as 3D-aware supervision for 3D creation. Our method presents the first attempt to achieve high-quality 3D creation from a single image for general objects and enables various applications such as text-to-3D creation and texture editing.
  arXiv  Detail & Related papers  (2023-03-24T17:54:22Z)
• 3DAvatarGAN: Bridging Domains for Personalized Editable Avatars [75.31960120109106]
  3D-GANs synthesize geometry and texture by training on large-scale datasets with a consistent structure. We propose an adaptation framework, where the source domain is a pre-trained 3D-GAN, while the target domain is a 2D-GAN trained on artistic datasets. We show a deformation-based technique for modeling exaggerated geometry of artistic domains, enabling -- as a byproduct -- personalized geometric editing.
  arXiv  Detail & Related papers  (2023-01-06T19:58:47Z)
• Towards Realistic 3D Embedding via View Alignment [53.89445873577063]
  This paper presents an innovative View Alignment GAN (VA-GAN) that composes new images by embedding 3D models into 2D background images realistically and automatically. VA-GAN consists of a texture generator and a differential discriminator that are inter-connected and end-to-end trainable.
  arXiv  Detail & Related papers  (2020-07-14T14:45:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.