Style-NeRF2NeRF: 3D Style Transfer From Style-Aligned Multi-View Images

• URL: http://arxiv.org/abs/2406.13393v3
• Date: Wed, 4 Sep 2024 06:32:00 GMT
• Title: Style-NeRF2NeRF: 3D Style Transfer From Style-Aligned Multi-View Images
• Authors: Haruo Fujiwara, Yusuke Mukuta, Tatsuya Harada,
• Abstract summary: We propose a simple yet effective pipeline for stylizing a 3D scene. We perform 3D style transfer by refining the source NeRF model using stylized images generated by a style-aligned image-to-image diffusion model. We demonstrate that our method can transfer diverse artistic styles to real-world 3D scenes with competitive quality.
• Score: 54.56070204172398
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a simple yet effective pipeline for stylizing a 3D scene, harnessing the power of 2D image diffusion models. Given a NeRF model reconstructed from a set of multi-view images, we perform 3D style transfer by refining the source NeRF model using stylized images generated by a style-aligned image-to-image diffusion model. Given a target style prompt, we first generate perceptually similar multi-view images by leveraging a depth-conditioned diffusion model with an attention-sharing mechanism. Next, based on the stylized multi-view images, we propose to guide the style transfer process with the sliced Wasserstein loss based on the feature maps extracted from a pre-trained CNN model. Our pipeline consists of decoupled steps, allowing users to test various prompt ideas and preview the stylized 3D result before proceeding to the NeRF fine-tuning stage. We demonstrate that our method can transfer diverse artistic styles to real-world 3D scenes with competitive quality. Result videos are also available on our project page: https://haruolabs.github.io/style-n2n/

Related papers

• Towards Multi-View Consistent Style Transfer with One-Step Diffusion via Vision Conditioning [12.43848969320173]
  Stylized images from different viewpoints generated by our method achieve superior visual quality, with better structural integrity and less distortion. Our method effectively preserves the structural information and multi-view consistency in stylized images without any 3D information.
  arXiv  Detail & Related papers  (2024-11-15T12:02:07Z)
• G3DST: Generalizing 3D Style Transfer with Neural Radiance Fields across Scenes and Styles [45.92812062685523]
  Existing methods for 3D style transfer need extensive per-scene optimization for single or multiple styles. In this work, we overcome the limitations of existing methods by rendering stylized novel views from a NeRF without the need for per-scene or per-style optimization. Our findings demonstrate that this approach achieves a good visual quality comparable to that of per-scene methods.
  arXiv  Detail & Related papers  (2024-08-24T08:04:19Z)
• NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows [60.291277312569285]
  We present a method for automatically modifying a NeRF representation based on a single observation. Our method defines the transformation as a 3D flow, specifically as a weighted linear blending of rigid transformations. We also introduce a new dataset for exploring the problem of modifying a NeRF scene through a single observation.
  arXiv  Detail & Related papers  (2024-06-15T07:58:08Z)
• Envision3D: One Image to 3D with Anchor Views Interpolation [18.31796952040799]
  We present Envision3D, a novel method for efficiently generating high-quality 3D content from a single image. It is capable of generating high-quality 3D content in terms of texture and geometry, surpassing previous image-to-3D baseline methods.
  arXiv  Detail & Related papers  (2024-03-13T18:46:33Z)
• 3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models [102.75875255071246]
  3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. We propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models.
  arXiv  Detail & Related papers  (2023-11-09T15:51:27Z)
• ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections [71.46546520120162]
  Estimating 3D articulated shapes like animal bodies from monocular images is inherently challenging. We propose ARTIC3D, a self-supervised framework to reconstruct per-instance 3D shapes from a sparse image collection in-the-wild. We produce realistic animations by fine-tuning the rendered shape and texture under rigid part transformations.
  arXiv  Detail & Related papers  (2023-06-07T17:47:50Z)
• StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning [50.65015652968839]
  3D scene stylization aims at generating stylized images of the scene from arbitrary novel views. Thanks to recently proposed neural radiance fields (NeRF), we are able to represent a 3D scene in a consistent way. We propose a novel mutual learning framework for 3D scene stylization that combines a 2D image stylization network and NeRF.
  arXiv  Detail & Related papers  (2022-05-24T16:29:50Z)
• 3DSNet: Unsupervised Shape-to-Shape 3D Style Transfer [66.48720190245616]
  We propose a learning-based approach for style transfer between 3D objects. The proposed method can synthesize new 3D shapes both in the form of point clouds and meshes. We extend our technique to implicitly learn the multimodal style distribution of the chosen domains.
  arXiv  Detail & Related papers  (2020-11-26T16:59:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.