Universal Photorealistic Style Transfer: A Lightweight and Adaptive Approach

• URL: http://arxiv.org/abs/2309.10011v3
• Date: Wed, 20 Nov 2024 23:27:25 GMT
• Title: Universal Photorealistic Style Transfer: A Lightweight and Adaptive Approach
• Authors: Rong Liu, Enyu Zhao, Zhiyuan Liu, Andrew Feng, Scott John Easley,
• Abstract summary: Photorealistic style transfer aims to apply stylization while preserving the realism and structure of input content. We propose a Universal Photorealistic Style Transfer framework that delivers accurate photorealistic style transfer on high-resolution images and videos.
• Score: 18.46446576915766
• License:
• Abstract: Photorealistic style transfer aims to apply stylization while preserving the realism and structure of input content. However, existing methods often encounter challenges such as color tone distortions, dependency on pair-wise pre-training, inefficiency with high-resolution inputs, and the need for additional constraints in video style transfer tasks. To address these issues, we propose a Universal Photorealistic Style Transfer (UPST) framework that delivers accurate photorealistic style transfer on high-resolution images and videos without relying on pre-training. Our approach incorporates a lightweight StyleNet for per-instance transfer, ensuring color tone accuracy while supporting high-resolution inputs, maintaining rapid processing speeds, and eliminating the need for pretraining. To further enhance photorealism and efficiency, we introduce instance-adaptive optimization, which features an adaptive coefficient to prioritize content image realism and employs early stopping to accelerate network convergence. Additionally, UPST enables seamless video style transfer without additional constraints due to its strong non-color information preservation ability. Experimental results show that UPST consistently produces photorealistic outputs and significantly reduces GPU memory usage, making it an effective and universal solution for various photorealistic style transfer tasks.

Related papers

• UniVST: A Unified Framework for Training-free Localized Video Style Transfer [66.69471376934034]
  This paper presents UniVST, a unified framework for localized video style transfer. It operates without the need for training, offering a distinct advantage over existing methods that transfer style across entire videos.
  arXiv  Detail & Related papers  (2024-10-26T05:28:02Z)
• ZePo: Zero-Shot Portrait Stylization with Faster Sampling [61.14140480095604]
  This paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control.
  arXiv  Detail & Related papers  (2024-08-10T08:53:41Z)
• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• Alignment-free HDR Deghosting with Semantics Consistent Transformer [76.91669741684173]
  High dynamic range imaging aims to retrieve information from multiple low-dynamic range inputs to generate realistic output. Existing methods often focus on the spatial misalignment across input frames caused by the foreground and/or camera motion. We propose a novel alignment-free network with a Semantics Consistent Transformer (SCTNet) with both spatial and channel attention modules.
  arXiv  Detail & Related papers  (2023-05-29T15:03:23Z)
• ColoristaNet for Photorealistic Video Style Transfer [15.38024996795316]
  Photorealistic style transfer aims to transfer the artistic style of an image onto an input image or video while keeping photorealism. We propose a self-supervised style transfer framework, which contains a style removal part and a style restoration part. Experiments demonstrate that ColoristaNet can achieve better stylization effects when compared with state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2022-12-19T04:49:26Z)
• CCPL: Contrastive Coherence Preserving Loss for Versatile Style Transfer [58.020470877242865]
  We devise a universally versatile style transfer method capable of performing artistic, photo-realistic, and video style transfer jointly. We make a mild and reasonable assumption that global inconsistency is dominated by local inconsistencies and devise a generic Contrastive Coherence Preserving Loss (CCPL) applied to local patches. CCPL can preserve the coherence of the content source during style transfer without degrading stylization.
  arXiv  Detail & Related papers  (2022-07-11T12:09:41Z)
• Deep Translation Prior: Test-time Training for Photorealistic Style Transfer [36.82737412912885]
  Recent techniques to solve photorealistic style transfer within deep convolutional neural networks (CNNs) generally require intensive training from large-scale datasets. We propose a novel framework, dubbed Deep Translation Prior (DTP), to accomplish photorealistic style transfer through test-time training on given input image pair with untrained networks.
  arXiv  Detail & Related papers  (2021-12-12T04:54:27Z)
• Controllable Person Image Synthesis with Spatially-Adaptive Warped Normalization [72.65828901909708]
  Controllable person image generation aims to produce realistic human images with desirable attributes. We introduce a novel Spatially-Adaptive Warped Normalization (SAWN), which integrates a learned flow-field to warp modulation parameters. We propose a novel self-training part replacement strategy to refine the pretrained model for the texture-transfer task.
  arXiv  Detail & Related papers  (2021-05-31T07:07:44Z)
• Joint Bilateral Learning for Real-time Universal Photorealistic Style Transfer [18.455002563426262]
  Photorealistic style transfer is the task of transferring the artistic style of an image onto a content target, producing a result that is plausibly taken with a camera. Recent approaches, based on deep neural networks, produce impressive results but are either too slow to run at practical resolutions, or still contain objectionable artifacts. We propose a new end-to-end model for photorealistic style transfer that is both fast and inherently generates photorealistic results.
  arXiv  Detail & Related papers  (2020-04-23T03:31:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.