Realtime Fewshot Portrait Stylization Based On Geometric Alignment
- URL: http://arxiv.org/abs/2211.15549v1
- Date: Mon, 28 Nov 2022 16:53:19 GMT
- Title: Realtime Fewshot Portrait Stylization Based On Geometric Alignment
- Authors: Xinrui Wang, Zhuoru Li, Xiao Zhou, Yusuke Iwasawa, Yutaka Matsuo
- Abstract summary: This paper presents a portrait stylization method designed for real-time mobile applications with limited style examples available.
Previous learning based stylization methods suffer from the geometric and semantic gaps between portrait domain and style domain.
Based on the geometric prior of human facial attributions, we propose to utilize geometric alignment to tackle this issue.
- Score: 32.224973317381426
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: This paper presents a portrait stylization method designed for real-time
mobile applications with limited style examples available. Previous learning
based stylization methods suffer from the geometric and semantic gaps between
portrait domain and style domain, which obstacles the style information to be
correctly transferred to the portrait images, leading to poor stylization
quality. Based on the geometric prior of human facial attributions, we propose
to utilize geometric alignment to tackle this issue. Firstly, we apply
Thin-Plate-Spline (TPS) on feature maps in the generator network and also
directly to style images in pixel space, generating aligned portrait-style
image pairs with identical landmarks, which closes the geometric gaps between
two domains. Secondly, adversarial learning maps the textures and colors of
portrait images to the style domain. Finally, geometric aware cycle consistency
preserves the content and identity information unchanged, and deformation
invariant constraint suppresses artifacts and distortions. Qualitative and
quantitative comparison validate our method outperforms existing methods, and
experiments proof our method could be trained with limited style examples (100
or less) in real-time (more than 40 FPS) on mobile devices. Ablation study
demonstrates the effectiveness of each component in the framework.
Related papers
- Customizing Text-to-Image Models with a Single Image Pair [47.49970731632113]
Art reinterpretation is the practice of creating a variation of a reference work, making a paired artwork that exhibits a distinct artistic style.
We propose Pair Customization, a new customization method that learns stylistic difference from a single image pair and then applies the acquired style to the generation process.
arXiv Detail & Related papers (2024-05-02T17:59:52Z) - InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation [5.364489068722223]
The concept of style is inherently underdetermined, encompassing a multitude of elements such as color, material, atmosphere, design, and structure.
Inversion-based methods are prone to style degradation, often resulting in the loss of fine-grained details.
adapter-based approaches frequently require meticulous weight tuning for each reference image to achieve a balance between style intensity and text controllability.
arXiv Detail & Related papers (2024-04-03T13:34:09Z) - Portrait Diffusion: Training-free Face Stylization with
Chain-of-Painting [64.43760427752532]
Face stylization refers to the transformation of a face into a specific portrait style.
Current methods require the use of example-based adaptation approaches to fine-tune pre-trained generative models.
This paper proposes a training-free face stylization framework, named Portrait Diffusion.
arXiv Detail & Related papers (2023-12-03T06:48:35Z) - A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive
Learning [84.8813842101747]
Unified Contrastive Arbitrary Style Transfer (UCAST) is a novel style representation learning and transfer framework.
We present an adaptive contrastive learning scheme for style transfer by introducing an input-dependent temperature.
Our framework consists of three key components, i.e., a parallel contrastive learning scheme for style representation and style transfer, a domain enhancement module for effective learning of style distribution, and a generative network for style transfer.
arXiv Detail & Related papers (2023-03-09T04:35:00Z) - Parallax-Tolerant Unsupervised Deep Image Stitching [57.76737888499145]
We propose UDIS++, a parallax-tolerant unsupervised deep image stitching technique.
First, we propose a robust and flexible warp to model the image registration from global homography to local thin-plate spline motion.
To further eliminate the parallax artifacts, we propose to composite the stitched image seamlessly by unsupervised learning for seam-driven composition masks.
arXiv Detail & Related papers (2023-02-16T10:40:55Z) - Learning Graph Neural Networks for Image Style Transfer [131.73237185888215]
State-of-the-art parametric and non-parametric style transfer approaches are prone to either distorted local style patterns due to global statistics alignment, or unpleasing artifacts resulting from patch mismatching.
In this paper, we study a novel semi-parametric neural style transfer framework that alleviates the deficiency of both parametric and non-parametric stylization.
arXiv Detail & Related papers (2022-07-24T07:41:31Z) - Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning [84.8813842101747]
Contrastive Arbitrary Style Transfer (CAST) is a new style representation learning and style transfer method via contrastive learning.
Our framework consists of three key components, i.e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer.
arXiv Detail & Related papers (2022-05-19T13:11:24Z) - Controllable Person Image Synthesis with Spatially-Adaptive Warped
Normalization [72.65828901909708]
Controllable person image generation aims to produce realistic human images with desirable attributes.
We introduce a novel Spatially-Adaptive Warped Normalization (SAWN), which integrates a learned flow-field to warp modulation parameters.
We propose a novel self-training part replacement strategy to refine the pretrained model for the texture-transfer task.
arXiv Detail & Related papers (2021-05-31T07:07:44Z) - Deformable Style Transfer [14.729482749508374]
We propose deformable style transfer (DST), an optimization-based approach that jointly stylizes the texture and geometry of a content image to better match a style image.
We demonstrate our method on a diverse set of content and style images including portraits, animals, objects, scenes, and paintings.
arXiv Detail & Related papers (2020-03-24T18:00:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.