SNeRF: Stylized Neural Implicit Representations for 3D Scenes

• URL: http://arxiv.org/abs/2207.02363v1
• Date: Tue, 5 Jul 2022 23:45:02 GMT
• Title: SNeRF: Stylized Neural Implicit Representations for 3D Scenes
• Authors: Thu Nguyen-Phuoc, Feng Liu, Lei Xiao
• Abstract summary: This paper investigates 3D scene stylization that provides a strong inductive bias for consistent novel view synthesis. We adopt the emerging neural radiance fields (NeRF) as our choice of 3D scene representation. We introduce a new training method to address this problem by alternating the NeRF and stylization optimization steps.
• Score: 9.151746397358522
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper presents a stylized novel view synthesis method. Applying state-of-the-art stylization methods to novel views frame by frame often causes jittering artifacts due to the lack of cross-view consistency. Therefore, this paper investigates 3D scene stylization that provides a strong inductive bias for consistent novel view synthesis. Specifically, we adopt the emerging neural radiance fields (NeRF) as our choice of 3D scene representation for their capability to render high-quality novel views for a variety of scenes. However, as rendering a novel view from a NeRF requires a large number of samples, training a stylized NeRF requires a large amount of GPU memory that goes beyond an off-the-shelf GPU capacity. We introduce a new training method to address this problem by alternating the NeRF and stylization optimization steps. Such a method enables us to make full use of our hardware memory capacity to both generate images at higher resolution and adopt more expressive image style transfer methods. Our experiments show that our method produces stylized NeRFs for a wide range of content, including indoor, outdoor and dynamic scenes, and synthesizes high-quality novel views with cross-view consistency.

Related papers

• ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis [63.169364481672915]
  We propose textbfViewCrafter, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames.
  arXiv  Detail & Related papers  (2024-09-03T16:53:19Z)
• G3DST: Generalizing 3D Style Transfer with Neural Radiance Fields across Scenes and Styles [45.92812062685523]
  Existing methods for 3D style transfer need extensive per-scene optimization for single or multiple styles. In this work, we overcome the limitations of existing methods by rendering stylized novel views from a NeRF without the need for per-scene or per-style optimization. Our findings demonstrate that this approach achieves a good visual quality comparable to that of per-scene methods.
  arXiv  Detail & Related papers  (2024-08-24T08:04:19Z)
• ArtNeRF: A Stylized Neural Field for 3D-Aware Cartoonized Face Synthesis [11.463969116010183]
  ArtNeRF is a novel face stylization framework derived from 3D-aware GAN. We propose an expressive generator to synthesize stylized faces and a triple-branch discriminator module to improve style consistency. Experiments demonstrate that ArtNeRF is versatile in generating high-quality 3D-aware cartoon faces with arbitrary styles.
  arXiv  Detail & Related papers  (2024-04-21T16:45:35Z)
• Towards 4D Human Video Stylization [56.33756124829298]
  We present a first step towards 4D (3D and time) human video stylization, which addresses style transfer, novel view synthesis and human animation. We leverage Neural Radiance Fields (NeRFs) to represent videos, conducting stylization in the rendered feature space. Our framework uniquely extends its capabilities to accommodate novel poses and viewpoints, making it a versatile tool for creative human video stylization.
  arXiv  Detail & Related papers  (2023-12-07T08:58:33Z)
• HDHumans: A Hybrid Approach for High-fidelity Digital Humans [107.19426606778808]
  HDHumans is the first method for HD human character synthesis that jointly produces an accurate and temporally coherent 3D deforming surface. Our method is carefully designed to achieve a synergy between classical surface deformation and neural radiance fields (NeRF)
  arXiv  Detail & Related papers  (2022-10-21T14:42:11Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation [58.16911861917018]
  We present a novel method for performing flexible, 3D-aware image content manipulation while enabling high-quality novel view synthesis. Our model couples learnt scene-specific feature volumes with a scene agnostic neural rendering network. We demonstrate various scene manipulations, including mixing scenes, deforming objects and inserting objects into scenes, while still producing photo-realistic results.
  arXiv  Detail & Related papers  (2022-04-22T17:57:00Z)
• NeuralMVS: Bridging Multi-View Stereo and Novel View Synthesis [28.83180559337126]
  We propose a novel network that can recover 3D scene geometry as a distance function, together with high-resolution color images. Our method uses only a sparse set of images as input and can generalize well to novel scenes.
  arXiv  Detail & Related papers  (2021-08-09T08:59:24Z)
• Stylizing 3D Scene via Implicit Representation and HyperNetwork [34.22448260525455]
  A straightforward solution is to combine existing novel view synthesis and image/video style transfer approaches. Inspired by the high quality results of the neural radiance fields (NeRF) method, we propose a joint framework to directly render novel views with the desired style. Our framework consists of two components: an implicit representation of the 3D scene with the neural radiance field model, and a hypernetwork to transfer the style information into the scene representation.
  arXiv  Detail & Related papers  (2021-05-27T09:11:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.