NLUT: Neural-based 3D Lookup Tables for Video Photorealistic Style Transfer

• URL: http://arxiv.org/abs/2303.09170v2
• Date: Fri, 17 Mar 2023 05:10:48 GMT
• Title: NLUT: Neural-based 3D Lookup Tables for Video Photorealistic Style Transfer
• Authors: Yaosen Chen, Han Yang, Yuexin Yang, Yuegen Liu, Wei Wang, Xuming Wen, Chaoping Xie
• Abstract summary: Video style transfer is desired to generate with a similar photorealistic style to the style image while maintaining temporal consistency. Existing methods obtain stylized video sequences by performing frame-by-frame photorealistic style transfer, which is inefficient and does not ensure the temporal consistency of the stylized video. We first train a neural network for generating stylized 3D LUTs on a large-scale dataset; then, when performing photorealistic style transfer for a specific video, we select a videos and style image in the video as the data source and fine-turn the neural network. Finally, we query the 3D LUTs generated by the fine-
• Score: 5.442253227842167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video photorealistic style transfer is desired to generate videos with a similar photorealistic style to the style image while maintaining temporal consistency. However, existing methods obtain stylized video sequences by performing frame-by-frame photorealistic style transfer, which is inefficient and does not ensure the temporal consistency of the stylized video. To address this issue, we use neural network-based 3D Lookup Tables (LUTs) for the photorealistic transfer of videos, achieving a balance between efficiency and effectiveness. We first train a neural network for generating photorealistic stylized 3D LUTs on a large-scale dataset; then, when performing photorealistic style transfer for a specific video, we select a keyframe and style image in the video as the data source and fine-turn the neural network; finally, we query the 3D LUTs generated by the fine-tuned neural network for the colors in the video, resulting in a super-fast photorealistic style transfer, even processing 8K video takes less than 2 millisecond per frame. The experimental results show that our method not only realizes the photorealistic style transfer of arbitrary style images but also outperforms the existing methods in terms of visual quality and consistency. Project page:https://semchan.github.io/NLUT_Project.

Related papers

• Tex4D: Zero-shot 4D Scene Texturing with Video Diffusion Models [54.35214051961381]
  3D meshes are widely used in computer vision and graphics for their efficiency in animation and minimal memory use in movies, games, AR, and VR. However, creating temporal consistent and realistic textures for mesh remains labor-intensive for professional artists. We present 3D Tex sequences that integrates inherent geometry from mesh sequences with video diffusion models to produce consistent textures.
  arXiv  Detail & Related papers  (2024-10-14T17:59:59Z)
• Style-NeRF2NeRF: 3D Style Transfer From Style-Aligned Multi-View Images [54.56070204172398]
  We propose a simple yet effective pipeline for stylizing a 3D scene. We perform 3D style transfer by refining the source NeRF model using stylized images generated by a style-aligned image-to-image diffusion model. We demonstrate that our method can transfer diverse artistic styles to real-world 3D scenes with competitive quality.
  arXiv  Detail & Related papers  (2024-06-19T09:36:18Z)
• StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting [141.05924680451804]
  StyleGaussian is a novel 3D style transfer technique. It allows instant transfer of any image's style to a 3D scene at 10 frames per second (fps)
  arXiv  Detail & Related papers  (2024-03-12T16:44:52Z)
• WAIT: Feature Warping for Animation to Illustration video Translation using GANs [12.681919619814419]
  We introduce a new problem for video stylizing where an unordered set of images are used. Most of the video-to-video translation methods are built on an image-to-image translation model. We propose a new generator network with feature warping layers which overcomes the limitations of the previous methods.
  arXiv  Detail & Related papers  (2023-10-07T19:45:24Z)
• NILUT: Conditional Neural Implicit 3D Lookup Tables for Image Enhancement [82.75363196702381]
  3D lookup tables (3D LUTs) are a key component for image enhancement. Current approaches for learning and applying 3D LUTs are notably fast, yet not so memory-efficient. We propose a Neural Implicit LUT (NILUT), an implicitly defined continuous 3D color transformation parameterized by a neural network.
  arXiv  Detail & Related papers  (2023-06-20T22:06:39Z)
• UPST-NeRF: Universal Photorealistic Style Transfer of Neural Radiance Fields for 3D Scene [2.1033122829097484]
  3D scenes photorealistic stylization aims to generate photorealistic images from arbitrary novel views according to a given style image. Some existing stylization methods with neural radiance fields can effectively predict stylized scenes. We propose a novel 3D scene photorealistic style transfer framework to address these issues.
  arXiv  Detail & Related papers  (2022-08-15T08:17:35Z)
• Video2StyleGAN: Encoding Video in Latent Space for Manipulation [63.03250800510085]
  We propose a novel network to encode face videos into the latent space of StyleGAN for semantic face video manipulation. Our approach can significantly outperform existing single image methods, while achieving real-time (66 fps) speed.
  arXiv  Detail & Related papers  (2022-06-27T06:48:15Z)
• Robust Pose Transfer with Dynamic Details using Neural Video Rendering [48.48929344349387]
  We propose a neural video rendering framework coupled with an image-translation-based dynamic details generation network (D2G-Net) To be specific, a novel texture representation is presented to encode both the static and pose-varying appearance characteristics. We demonstrate that our neural human video is capable of achieving both clearer dynamic details and more robust performance even on short videos with only 2k - 4k frames.
  arXiv  Detail & Related papers  (2021-06-27T03:40:22Z)
• Joint Bilateral Learning for Real-time Universal Photorealistic Style Transfer [18.455002563426262]
  Photorealistic style transfer is the task of transferring the artistic style of an image onto a content target, producing a result that is plausibly taken with a camera. Recent approaches, based on deep neural networks, produce impressive results but are either too slow to run at practical resolutions, or still contain objectionable artifacts. We propose a new end-to-end model for photorealistic style transfer that is both fast and inherently generates photorealistic results.
  arXiv  Detail & Related papers  (2020-04-23T03:31:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.