Third Time's the Charm? Image and Video Editing with StyleGAN3

• URL: http://arxiv.org/abs/2201.13433v1
• Date: Mon, 31 Jan 2022 18:44:59 GMT
• Title: Third Time's the Charm? Image and Video Editing with StyleGAN3
• Authors: Yuval Alaluf, Or Patashnik, Zongze Wu, Asif Zamir, Eli Shechtman, Dani Lischinski, Daniel Cohen-Or
• Abstract summary: StyleGAN is arguably one of the most intriguing and well-studied generative models. We explore the recent StyleGAN3 architecture, compare it to its predecessor, and investigate its unique advantages, as well as drawbacks.
• Score: 70.36056009463738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: StyleGAN is arguably one of the most intriguing and well-studied generative models, demonstrating impressive performance in image generation, inversion, and manipulation. In this work, we explore the recent StyleGAN3 architecture, compare it to its predecessor, and investigate its unique advantages, as well as drawbacks. In particular, we demonstrate that while StyleGAN3 can be trained on unaligned data, one can still use aligned data for training, without hindering the ability to generate unaligned imagery. Next, our analysis of the disentanglement of the different latent spaces of StyleGAN3 indicates that the commonly used W/W+ spaces are more entangled than their StyleGAN2 counterparts, underscoring the benefits of using the StyleSpace for fine-grained editing. Considering image inversion, we observe that existing encoder-based techniques struggle when trained on unaligned data. We therefore propose an encoding scheme trained solely on aligned data, yet can still invert unaligned images. Finally, we introduce a novel video inversion and editing workflow that leverages the capabilities of a fine-tuned StyleGAN3 generator to reduce texture sticking and expand the field of view of the edited video.

Related papers

• WaSt-3D: Wasserstein-2 Distance for Scene-to-Scene Stylization on 3D Gaussians [37.139479729087896]
  We develop a new style transfer method for 3D scenes called WaSt-3D. It faithfully transfers details from style scenes to the content scene without requiring any training. WaSt-3D consistently delivers results across diverse content and style scenes without necessitating any training.
  arXiv  Detail & Related papers  (2024-09-26T15:02:50Z)
• InstantStyleGaussian: Efficient Art Style Transfer with 3D Gaussian Splatting [1.495965529797126]
  We present InstantStyleGaussian, an innovative 3D style transfer method based on the 3D Gaussian Splatting (3DGS) scene representation. By inputting a target-style image, it quickly generates new 3D GS scenes.
  arXiv  Detail & Related papers  (2024-08-08T06:29:32Z)
• Reference-based Controllable Scene Stylization with Gaussian Splatting [30.321151430263946]
  Referenced-based scene stylization that edits the appearance based on a content-aligned reference image is an emerging research area. We propose ReGS, which adapts 3D Gaussian Splatting (3DGS) for reference-based stylization to enable real-time stylized view synthesis.
  arXiv  Detail & Related papers  (2024-07-09T20:30:29Z)
• StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting [141.05924680451804]
  StyleGaussian is a novel 3D style transfer technique. It allows instant transfer of any image's style to a 3D scene at 10 frames per second (fps)
  arXiv  Detail & Related papers  (2024-03-12T16:44:52Z)
• Learning Naturally Aggregated Appearance for Efficient 3D Editing [94.47518916521065]
  We propose to replace the color field with an explicit 2D appearance aggregation, also called canonical image. To avoid the distortion effect and facilitate convenient editing, we complement the canonical image with a projection field that maps 3D points onto 2D pixels for texture lookup. Our representation, dubbed AGAP, well supports various ways of 3D editing (e.g., stylization, interactive drawing, and content extraction) with no need of re-optimization.
  arXiv  Detail & Related papers  (2023-12-11T18:59:31Z)
• DeformToon3D: Deformable 3D Toonification from Neural Radiance Fields [96.0858117473902]
  3D toonification involves transferring the style of an artistic domain onto a target 3D face with stylized geometry and texture. We propose DeformToon3D, an effective toonification framework tailored for hierarchical 3D GAN. Our approach decomposes 3D toonification into subproblems of geometry and texture stylization to better preserve the original latent space.
  arXiv  Detail & Related papers  (2023-09-08T16:17:45Z)
• NeRFEditor: Differentiable Style Decomposition for Full 3D Scene Editing [37.06344045938838]
  We present NeRFEditor, an efficient learning framework for 3D scene editing. NeRFEditor takes a video captured over 360deg as input and outputs a high-quality, identity-preserving stylized 3D scene.
  arXiv  Detail & Related papers  (2022-12-07T18:44:28Z)
• ARF: Artistic Radiance Fields [63.79314417413371]
  We present a method for transferring the artistic features of an arbitrary style image to a 3D scene. Previous methods that perform 3D stylization on point clouds or meshes are sensitive to geometric reconstruction errors. We propose to stylize the more robust radiance field representation.
  arXiv  Detail & Related papers  (2022-06-13T17:55:31Z)
• StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions [11.153966202832933]
  We apply style transfer on mesh reconstructions of indoor scenes. This enables VR applications like experiencing 3D environments painted in the style of a favorite artist.
  arXiv  Detail & Related papers  (2021-12-02T18:59:59Z)
• Learning to Stylize Novel Views [82.24095446809946]
  We tackle a 3D scene stylization problem - generating stylized images of a scene from arbitrary novel views. We propose a point cloud-based method for consistent 3D scene stylization.
  arXiv  Detail & Related papers  (2021-05-27T23:58:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.