ArtNeRF: A Stylized Neural Field for 3D-Aware Cartoonized Face Synthesis

• URL: http://arxiv.org/abs/2404.13711v2
• Date: Fri, 26 Apr 2024 02:53:52 GMT
• Title: ArtNeRF: A Stylized Neural Field for 3D-Aware Cartoonized Face Synthesis
• Authors: Zichen Tang, Hongyu Yang,
• Abstract summary: ArtNeRF is a novel face stylization framework derived from 3D-aware GAN. We propose an expressive generator to synthesize stylized faces and a triple-branch discriminator module to improve style consistency. Experiments demonstrate that ArtNeRF is versatile in generating high-quality 3D-aware cartoon faces with arbitrary styles.
• Score: 11.463969116010183
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in generative visual models and neural radiance fields have greatly boosted 3D-aware image synthesis and stylization tasks. However, previous NeRF-based work is limited to single scene stylization, training a model to generate 3D-aware cartoon faces with arbitrary styles remains unsolved. We propose ArtNeRF, a novel face stylization framework derived from 3D-aware GAN to tackle this problem. In this framework, we utilize an expressive generator to synthesize stylized faces and a triple-branch discriminator module to improve the visual quality and style consistency of the generated faces. Specifically, a style encoder based on contrastive learning is leveraged to extract robust low-dimensional embeddings of style images, empowering the generator with the knowledge of various styles. To smooth the training process of cross-domain transfer learning, we propose an adaptive style blending module which helps inject style information and allows users to freely tune the level of stylization. We further introduce a neural rendering module to achieve efficient real-time rendering of images with higher resolutions. Extensive experiments demonstrate that ArtNeRF is versatile in generating high-quality 3D-aware cartoon faces with arbitrary styles.

Related papers

• G3DST: Generalizing 3D Style Transfer with Neural Radiance Fields across Scenes and Styles [45.92812062685523]
  Existing methods for 3D style transfer need extensive per-scene optimization for single or multiple styles. In this work, we overcome the limitations of existing methods by rendering stylized novel views from a NeRF without the need for per-scene or per-style optimization. Our findings demonstrate that this approach achieves a good visual quality comparable to that of per-scene methods.
  arXiv  Detail & Related papers  (2024-08-24T08:04:19Z)
• Style-NeRF2NeRF: 3D Style Transfer From Style-Aligned Multi-View Images [54.56070204172398]
  We propose a simple yet effective pipeline for stylizing a 3D scene. We perform 3D style transfer by refining the source NeRF model using stylized images generated by a style-aligned image-to-image diffusion model. We demonstrate that our method can transfer diverse artistic styles to real-world 3D scenes with competitive quality.
  arXiv  Detail & Related papers  (2024-06-19T09:36:18Z)
• Gaussian Splatting in Style [32.41970914897462]
  3D sceneization extends the work of neural style transfer to 3D. A vital challenge in this problem is to maintain the uniformity of the stylized appearance across multiple views. We propose a novel architecture trained on a collection of style images that, at test time, produces real time high-quality stylized novel views.
  arXiv  Detail & Related papers  (2024-03-13T13:06:31Z)
• Neural 3D Strokes: Creating Stylized 3D Scenes with Vectorized 3D Strokes [20.340259111585873]
  We present Neural 3D Strokes, a novel technique to generate stylized images of a 3D scene at arbitrary novel views from multi-view 2D images. Our approach draws inspiration from image-to-painting methods, simulating the progressive painting process of human artwork with vector strokes.
  arXiv  Detail & Related papers  (2023-11-27T09:02:21Z)
• DeformToon3D: Deformable 3D Toonification from Neural Radiance Fields [96.0858117473902]
  3D toonification involves transferring the style of an artistic domain onto a target 3D face with stylized geometry and texture. We propose DeformToon3D, an effective toonification framework tailored for hierarchical 3D GAN. Our approach decomposes 3D toonification into subproblems of geometry and texture stylization to better preserve the original latent space.
  arXiv  Detail & Related papers  (2023-09-08T16:17:45Z)
• HyperStyle3D: Text-Guided 3D Portrait Stylization via Hypernetworks [101.36230756743106]
  This paper is inspired by the success of 3D-aware GANs that bridge 2D and 3D domains with 3D fields as the intermediate representation for rendering 2D images. We propose a novel method, dubbed HyperStyle3D, based on 3D-aware GANs for 3D portrait stylization.
  arXiv  Detail & Related papers  (2023-04-19T07:22:05Z)
• StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields [52.19291190355375]
  StyleRF (Style Radiance Fields) is an innovative 3D style transfer technique. It employs an explicit grid of high-level features to represent 3D scenes, with which high-fidelity geometry can be reliably restored via volume rendering. It transforms the grid features according to the reference style which directly leads to high-quality zero-shot style transfer.
  arXiv  Detail & Related papers  (2023-03-19T08:26:06Z)
• NeRF-Art: Text-Driven Neural Radiance Fields Stylization [38.3724634394761]
  We present NeRF-Art, a text-guided NeRF stylization approach that manipulates the style of a pre-trained NeRF model with a simple text prompt. We show that our method is effective and robust regarding both single-view stylization quality and cross-view consistency.
  arXiv  Detail & Related papers  (2022-12-15T18:59:58Z)
• NeRFInvertor: High Fidelity NeRF-GAN Inversion for Single-shot Real Image Animation [66.0838349951456]
  Nerf-based Generative models have shown impressive capacity in generating high-quality images with consistent 3D geometry. We propose a universal method to surgically fine-tune these NeRF-GAN models in order to achieve high-fidelity animation of real subjects only by a single image.
  arXiv  Detail & Related papers  (2022-11-30T18:36:45Z)
• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis [92.25145204543904]
  StyleNeRF is a 3D-aware generative model for high-resolution image synthesis with high multi-view consistency. It integrates the neural radiance field (NeRF) into a style-based generator. It can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality.
  arXiv  Detail & Related papers  (2021-10-18T02:37:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.