Real-time Localized Photorealistic Video Style Transfer

• URL: http://arxiv.org/abs/2010.10056v1
• Date: Tue, 20 Oct 2020 06:21:09 GMT
• Title: Real-time Localized Photorealistic Video Style Transfer
• Authors: Xide Xia, Tianfan Xue, Wei-sheng Lai, Zheng Sun, Abby Chang, Brian Kulis, Jiawen Chen
• Abstract summary: We present a novel algorithm for transferring artistic styles of semantically meaningful local regions of an image onto local regions of a target video. Our method, based on a deep neural network architecture inspired by recent work in photorealistic style transfer, is real-time and works on arbitrary inputs. We demonstrate our method on a variety of style images and target videos, including the ability to transfer different styles onto multiple objects simultaneously.
• Score: 25.91181753178577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel algorithm for transferring artistic styles of semantically meaningful local regions of an image onto local regions of a target video while preserving its photorealism. Local regions may be selected either fully automatically from an image, through using video segmentation algorithms, or from casual user guidance such as scribbles. Our method, based on a deep neural network architecture inspired by recent work in photorealistic style transfer, is real-time and works on arbitrary inputs without runtime optimization once trained on a diverse dataset of artistic styles. By augmenting our video dataset with noisy semantic labels and jointly optimizing over style, content, mask, and temporal losses, our method can cope with a variety of imperfections in the input and produce temporally coherent videos without visual artifacts. We demonstrate our method on a variety of style images and target videos, including the ability to transfer different styles onto multiple objects simultaneously, and smoothly transition between styles in time.

Related papers

• Move Anything with Layered Scene Diffusion [77.45870343845492]
  We propose SceneDiffusion to optimize a layered scene representation during the diffusion sampling process. Our key insight is that spatial disentanglement can be obtained by jointly denoising scene renderings at different spatial layouts. Our generated scenes support a wide range of spatial editing operations, including moving, resizing, cloning, and layer-wise appearance editing operations.
  arXiv  Detail & Related papers  (2024-04-10T17:28:16Z)
• Towards 4D Human Video Stylization [56.33756124829298]
  We present a first step towards 4D (3D and time) human video stylization, which addresses style transfer, novel view synthesis and human animation. We leverage Neural Radiance Fields (NeRFs) to represent videos, conducting stylization in the rendered feature space. Our framework uniquely extends its capabilities to accommodate novel poses and viewpoints, making it a versatile tool for creative human video stylization.
  arXiv  Detail & Related papers  (2023-12-07T08:58:33Z)
• WAIT: Feature Warping for Animation to Illustration video Translation using GANs [12.681919619814419]
  We introduce a new problem for video stylizing where an unordered set of images are used. Most of the video-to-video translation methods are built on an image-to-image translation model. We propose a new generator network with feature warping layers which overcomes the limitations of the previous methods.
  arXiv  Detail & Related papers  (2023-10-07T19:45:24Z)
• Any-to-Any Style Transfer: Making Picasso and Da Vinci Collaborate [58.83278629019384]
  Style transfer aims to render the style of a given image for style reference to another given image for content reference. Existing approaches either apply the holistic style of the style image in a global manner, or migrate local colors and textures of the style image to the content counterparts in a pre-defined way. We propose Any-to-Any Style Transfer, which enables users to interactively select styles of regions in the style image and apply them to the prescribed content regions.
  arXiv  Detail & Related papers  (2023-04-19T15:15:36Z)
• Unsupervised Coherent Video Cartoonization with Perceptual Motion Consistency [89.75731026852338]
  We propose a spatially-adaptive alignment framework with perceptual motion consistency for coherent video cartoonization. We devise the semantic correlative map as a style-independent, global-aware regularization on the perceptual consistency motion. Our method is able to generate highly stylistic and temporal consistent cartoon videos.
  arXiv  Detail & Related papers  (2022-04-02T07:59:02Z)
• Interactive Style Transfer: All is Your Palette [74.06681967115594]
  We propose a drawing-like interactive style transfer (IST) method, by which users can interactively create a harmonious-style image. Our IST method can serve as a brush, dip style from anywhere, and then paint to any region of the target content image.
  arXiv  Detail & Related papers  (2022-03-25T06:38:46Z)
• STALP: Style Transfer with Auxiliary Limited Pairing [36.23393954839379]
  We present an approach to example-based stylization of images that uses a single pair of a source image and its stylized counterpart. We demonstrate how to train an image translation network that can perform real-time semantically meaningful style transfer to a set of target images.
  arXiv  Detail & Related papers  (2021-10-20T11:38:41Z)
• IBRNet: Learning Multi-View Image-Based Rendering [67.15887251196894]
  We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering.
  arXiv  Detail & Related papers  (2021-02-25T18:56:21Z)
• Interactive Video Stylization Using Few-Shot Patch-Based Training [23.801384381295698]
  We present a learning-based method to propagate the style from a few selecteds to the rest of the sequence. The resulting stylization is semantically meaningful, i.e., specific parts of moving objects are stylized according to the artist's intention. We demonstrate its practical utility in various interactive scenarios, where the user paints over a selected appearance and sees her style transferred to an existing recorded sequence or a live video stream.
  arXiv  Detail & Related papers  (2020-04-29T21:33:28Z)
• Joint Bilateral Learning for Real-time Universal Photorealistic Style Transfer [18.455002563426262]
  Photorealistic style transfer is the task of transferring the artistic style of an image onto a content target, producing a result that is plausibly taken with a camera. Recent approaches, based on deep neural networks, produce impressive results but are either too slow to run at practical resolutions, or still contain objectionable artifacts. We propose a new end-to-end model for photorealistic style transfer that is both fast and inherently generates photorealistic results.
  arXiv  Detail & Related papers  (2020-04-23T03:31:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.