Instant Photorealistic Style Transfer: A Lightweight and Adaptive
Approach
- URL: http://arxiv.org/abs/2309.10011v2
- Date: Fri, 20 Oct 2023 18:46:28 GMT
- Title: Instant Photorealistic Style Transfer: A Lightweight and Adaptive
Approach
- Authors: Rong Liu, Enyu Zhao, Zhiyuan Liu, Andrew Feng, Scott John Easley
- Abstract summary: We propose an Instant Photo Style Transfer (IPST) approach to achieve instant photorealistic style transfer on super-resolution inputs.
Our method utilizes a lightweight StyleNet to enable style transfer from a style image to a content image while preserving non-color information.
IPST is well-suited for multi-frame style transfer tasks, as it retains temporal and multi-view consistency of the multi-frame inputs.
- Score: 19.80952739530828
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper, we propose an Instant Photorealistic Style Transfer (IPST)
approach, designed to achieve instant photorealistic style transfer on
super-resolution inputs without the need for pre-training on pair-wise datasets
or imposing extra constraints. Our method utilizes a lightweight StyleNet to
enable style transfer from a style image to a content image while preserving
non-color information. To further enhance the style transfer process, we
introduce an instance-adaptive optimization to prioritize the photorealism of
outputs and accelerate the convergence of the style network, leading to a rapid
training completion within seconds. Moreover, IPST is well-suited for
multi-frame style transfer tasks, as it retains temporal and multi-view
consistency of the multi-frame inputs such as video and Neural Radiance Field
(NeRF). Experimental results demonstrate that IPST requires less GPU memory
usage, offers faster multi-frame transfer speed, and generates photorealistic
outputs, making it a promising solution for various photorealistic transfer
applications.
Related papers
- Ada-adapter:Fast Few-shot Style Personlization of Diffusion Model with Pre-trained Image Encoder [57.574544285878794]
Ada-Adapter is a novel framework for few-shot style personalization of diffusion models.
Our method enables efficient zero-shot style transfer utilizing a single reference image.
We demonstrate the effectiveness of our approach on various artistic styles, including flat art, 3D rendering, and logo design.
arXiv Detail & Related papers (2024-07-08T02:00:17Z) - Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability.
We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images.
Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
arXiv Detail & Related papers (2024-03-28T22:25:05Z) - Deep Translation Prior: Test-time Training for Photorealistic Style
Transfer [36.82737412912885]
Recent techniques to solve photorealistic style transfer within deep convolutional neural networks (CNNs) generally require intensive training from large-scale datasets.
We propose a novel framework, dubbed Deep Translation Prior (DTP), to accomplish photorealistic style transfer through test-time training on given input image pair with untrained networks.
arXiv Detail & Related papers (2021-12-12T04:54:27Z) - UMFA: A photorealistic style transfer method based on U-Net and
multi-layer feature aggregation [0.0]
We propose a photorealistic style transfer network to emphasize the natural effect of photorealistic image stylization.
In particular, an encoder based on the dense block and a decoder form a symmetrical structure of U-Net are jointly staked to realize an effective feature extraction and image reconstruction.
arXiv Detail & Related papers (2021-08-13T08:06:29Z) - StEP: Style-based Encoder Pre-training for Multi-modal Image Synthesis [68.3787368024951]
We propose a novel approach for multi-modal Image-to-image (I2I) translation.
We learn a latent embedding, jointly with the generator, that models the variability of the output domain.
Specifically, we pre-train a generic style encoder using a novel proxy task to learn an embedding of images, from arbitrary domains, into a low-dimensional style latent space.
arXiv Detail & Related papers (2021-04-14T19:58:24Z) - Spatially-Adaptive Pixelwise Networks for Fast Image Translation [57.359250882770525]
We introduce a new generator architecture, aimed at fast and efficient high-resolution image-to-image translation.
We use pixel-wise networks; that is, each pixel is processed independently of others.
Our model is up to 18x faster than state-of-the-art baselines.
arXiv Detail & Related papers (2020-12-05T10:02:03Z) - Real-time Universal Style Transfer on High-resolution Images via
Zero-channel Pruning [74.09149955786367]
ArtNet can achieve universal, real-time, and high-quality style transfer on high-resolution images simultaneously.
By using ArtNet and S2, our method is 2.3 to 107.4 times faster than state-of-the-art approaches.
arXiv Detail & Related papers (2020-06-16T09:50:14Z) - Learning Deformable Image Registration from Optimization: Perspective,
Modules, Bilevel Training and Beyond [62.730497582218284]
We develop a new deep learning based framework to optimize a diffeomorphic model via multi-scale propagation.
We conduct two groups of image registration experiments on 3D volume datasets including image-to-atlas registration on brain MRI data and image-to-image registration on liver CT data.
arXiv Detail & Related papers (2020-04-30T03:23:45Z) - Joint Bilateral Learning for Real-time Universal Photorealistic Style
Transfer [18.455002563426262]
Photorealistic style transfer is the task of transferring the artistic style of an image onto a content target, producing a result that is plausibly taken with a camera.
Recent approaches, based on deep neural networks, produce impressive results but are either too slow to run at practical resolutions, or still contain objectionable artifacts.
We propose a new end-to-end model for photorealistic style transfer that is both fast and inherently generates photorealistic results.
arXiv Detail & Related papers (2020-04-23T03:31:24Z) - Style Transfer for Light Field Photography [0.0]
It is necessary to adapt existing monocular style transfer networks in a way that allows for the stylization of each view of the light field.
The proposed method backpropagates the loss through the network, and the process is iterated to optimize the resulting stylization for a single light field image alone.
arXiv Detail & Related papers (2020-02-25T23:21:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.