StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning

• URL: http://arxiv.org/abs/2205.12183v2
• Date: Wed, 25 May 2022 05:19:33 GMT
• Title: StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning
• Authors: Yi-Hua Huang and Yue He and Yu-Jie Yuan and Yu-Kun Lai and Lin Gao
• Abstract summary: 3D scene stylization aims at generating stylized images of the scene from arbitrary novel views. Thanks to recently proposed neural radiance fields (NeRF), we are able to represent a 3D scene in a consistent way. We propose a novel mutual learning framework for 3D scene stylization that combines a 2D image stylization network and NeRF.
• Score: 50.65015652968839
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D scene stylization aims at generating stylized images of the scene from arbitrary novel views following a given set of style examples, while ensuring consistency when rendered from different views. Directly applying methods for image or video stylization to 3D scenes cannot achieve such consistency. Thanks to recently proposed neural radiance fields (NeRF), we are able to represent a 3D scene in a consistent way. Consistent 3D scene stylization can be effectively achieved by stylizing the corresponding NeRF. However, there is a significant domain gap between style examples which are 2D images and NeRF which is an implicit volumetric representation. To address this problem, we propose a novel mutual learning framework for 3D scene stylization that combines a 2D image stylization network and NeRF to fuse the stylization ability of 2D stylization network with the 3D consistency of NeRF. We first pre-train a standard NeRF of the 3D scene to be stylized and replace its color prediction module with a style network to obtain a stylized NeRF. It is followed by distilling the prior knowledge of spatial consistency from NeRF to the 2D stylization network through an introduced consistency loss. We also introduce a mimic loss to supervise the mutual learning of the NeRF style module and fine-tune the 2D stylization decoder. In order to further make our model handle ambiguities of 2D stylization results, we introduce learnable latent codes that obey the probability distributions conditioned on the style. They are attached to training samples as conditional inputs to better learn the style module in our novel stylized NeRF. Experimental results demonstrate that our method is superior to existing approaches in both visual quality and long-range consistency.

Related papers

• Style-NeRF2NeRF: 3D Style Transfer From Style-Aligned Multi-View Images [54.56070204172398]
  We propose a simple yet effective pipeline for stylizing a 3D scene. We perform 3D style transfer by refining the source NeRF model using stylized images generated by a style-aligned image-to-image diffusion model. We demonstrate that our method can transfer diverse artistic styles to real-world 3D scenes with competitive quality.
  arXiv  Detail & Related papers  (2024-06-19T09:36:18Z)
• PNeSM: Arbitrary 3D Scene Stylization via Prompt-Based Neural Style Mapping [16.506819625584654]
  3D scene stylization refers to transform the appearance of a 3D scene to match a given style image. Several existing methods have obtained impressive results in stylizing 3D scenes. We propose a novel 3D scene stylization framework to transfer an arbitrary style to an arbitrary scene.
  arXiv  Detail & Related papers  (2024-03-13T05:08:47Z)
• 3D Face Style Transfer with a Hybrid Solution of NeRF and Mesh Rasterization [4.668492532161309]
  We propose to use a neural radiance field (NeRF) to represent 3D human face and combine it with 2D style transfer to stylize the 3D face. We find that directly training a NeRF on stylized images from 2D style transfer brings in 3D inconsistency issue and causes blurriness. We propose a hybrid framework of NeRF and meshization to combine the benefits of high fidelity geometry reconstruction of NeRF and fast speed rendering of mesh.
  arXiv  Detail & Related papers  (2023-11-22T05:24:35Z)
• 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models [102.75875255071246]
  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. We propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
  arXiv  Detail & Related papers  (2023-11-09T15:51:27Z)
• HyperStyle3D: Text-Guided 3D Portrait Stylization via Hypernetworks [101.36230756743106]
  This paper is inspired by the success of 3D-aware GANs that bridge 2D and 3D domains with 3D fields as the intermediate representation for rendering 2D images. We propose a novel method, dubbed HyperStyle3D, based on 3D-aware GANs for 3D portrait stylization.
  arXiv  Detail & Related papers  (2023-04-19T07:22:05Z)
• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis [92.25145204543904]
  StyleNeRF is a 3D-aware generative model for high-resolution image synthesis with high multi-view consistency. It integrates the neural radiance field (NeRF) into a style-based generator. It can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality.
  arXiv  Detail & Related papers  (2021-10-18T02:37:01Z)
• Learning to Stylize Novel Views [82.24095446809946]
  We tackle a 3D scene stylization problem - generating stylized images of a scene from arbitrary novel views. We propose a point cloud-based method for consistent 3D scene stylization.
  arXiv  Detail & Related papers  (2021-05-27T23:58:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.