StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting

• URL: http://arxiv.org/abs/2403.07807v1
• Date: Tue, 12 Mar 2024 16:44:52 GMT
• Title: StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting
• Authors: Kunhao Liu, Fangneng Zhan, Muyu Xu, Christian Theobalt, Ling Shao, Shijian Lu
• Abstract summary: StyleGaussian is a novel 3D style transfer technique. It allows instant transfer of any image's style to a 3D scene at 10 frames per second (fps)
• Score: 141.05924680451804
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce StyleGaussian, a novel 3D style transfer technique that allows instant transfer of any image's style to a 3D scene at 10 frames per second (fps). Leveraging 3D Gaussian Splatting (3DGS), StyleGaussian achieves style transfer without compromising its real-time rendering ability and multi-view consistency. It achieves instant style transfer with three steps: embedding, transfer, and decoding. Initially, 2D VGG scene features are embedded into reconstructed 3D Gaussians. Next, the embedded features are transformed according to a reference style image. Finally, the transformed features are decoded into the stylized RGB. StyleGaussian has two novel designs. The first is an efficient feature rendering strategy that first renders low-dimensional features and then maps them into high-dimensional features while embedding VGG features. It cuts the memory consumption significantly and enables 3DGS to render the high-dimensional memory-intensive features. The second is a K-nearest-neighbor-based 3D CNN. Working as the decoder for the stylized features, it eliminates the 2D CNN operations that compromise strict multi-view consistency. Extensive experiments show that StyleGaussian achieves instant 3D stylization with superior stylization quality while preserving real-time rendering and strict multi-view consistency. Project page: https://kunhao-liu.github.io/StyleGaussian/

Related papers

• Gaussian Billboards: Expressive 2D Gaussian Splatting with Textures [8.724367699416893]
  We highlight the similarity between 2D Gaussian Splatting (2DGS) and billboards from traditional computer graphics. We propose a modification of 2DGS to add spatially-varying color achieved using per-splat texture. We show that our method can improve the sharpness and quality of the scene representation in a wide range of qualitative and quantitative evaluations.
  arXiv  Detail & Related papers  (2024-12-17T09:57:04Z)
• MiraGe: Editable 2D Images using Gaussian Splatting [0.0]
  Implicit Neural Representations (INRs) approximate discrete data through continuous functions and are commonly used for encoding 2D images. We introduce a novel method, MiraGe, which uses mirror reflections to perceive 2D images in 3D space and employs flat-controlled Gaussians for precise 2D image editing. Our approach improves the rendering quality and allows realistic image modifications, including human-inspired perception of photos in the 3D world.
  arXiv  Detail & Related papers  (2024-10-02T13:10:57Z)
• WaSt-3D: Wasserstein-2 Distance for Scene-to-Scene Stylization on 3D Gaussians [37.139479729087896]
  We develop a new style transfer method for 3D scenes called WaSt-3D. It faithfully transfers details from style scenes to the content scene without requiring any training. WaSt-3D consistently delivers results across diverse content and style scenes without necessitating any training.
  arXiv  Detail & Related papers  (2024-09-26T15:02:50Z)
• InstantStyleGaussian: Efficient Art Style Transfer with 3D Gaussian Splatting [1.495965529797126]
  We present InstantStyleGaussian, an innovative 3D style transfer method based on the 3D Gaussian Splatting (3DGS) scene representation. By inputting a target-style image, it quickly generates new 3D GS scenes.
  arXiv  Detail & Related papers  (2024-08-08T06:29:32Z)
• StyleSplat: 3D Object Style Transfer with Gaussian Splatting [0.3374875022248866]
  Style transfer can enhance 3D assets with diverse artistic styles, transforming creative expression. We introduce StyleSplat, a method for stylizing 3D objects in scenes represented by 3D Gaussians from reference style images. We demonstrate its effectiveness across various 3D scenes and styles, showcasing enhanced control and customization in 3D creation.
  arXiv  Detail & Related papers  (2024-07-12T17:55:08Z)
• Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation [55.73399465968594]
  This paper proposes a novel generation paradigm Sketch3D to generate realistic 3D assets with shape aligned with the input sketch and color matching the textual description. Three strategies are designed to optimize 3D Gaussians, i.e., structural optimization via a distribution transfer mechanism, color optimization with a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss.
  arXiv  Detail & Related papers  (2024-04-02T11:03:24Z)
• HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting [53.6394928681237]
  holistic understanding of urban scenes based on RGB images is a challenging yet important problem. Our main idea involves the joint optimization of geometry, appearance, semantics, and motion using a combination of static and dynamic 3D Gaussians. Our approach offers the ability to render new viewpoints in real-time, yielding 2D and 3D semantic information with high accuracy.
  arXiv  Detail & Related papers  (2024-03-19T13:39:05Z)
• Gaussian Shell Maps for Efficient 3D Human Generation [96.25056237689988]
  3D generative adversarial networks (GANs) have demonstrated state-of-the-art (SOTA) quality and diversity for generated assets. Current 3D GAN architectures, however, rely on volume representations, which are slow to render, thereby hampering the GAN training and requiring multi-view-inconsistent 2D upsamplers.
  arXiv  Detail & Related papers  (2023-11-29T18:04:07Z)
• 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models [102.75875255071246]
  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. We propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
  arXiv  Detail & Related papers  (2023-11-09T15:51:27Z)
• StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields [52.19291190355375]
  StyleRF (Style Radiance Fields) is an innovative 3D style transfer technique. It employs an explicit grid of high-level features to represent 3D scenes, with which high-fidelity geometry can be reliably restored via volume rendering. It transforms the grid features according to the reference style which directly leads to high-quality zero-shot style transfer.
  arXiv  Detail & Related papers  (2023-03-19T08:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.