Flexible Portrait Image Editing with Fine-Grained Control
- URL: http://arxiv.org/abs/2204.01318v1
- Date: Mon, 4 Apr 2022 08:39:37 GMT
- Title: Flexible Portrait Image Editing with Fine-Grained Control
- Authors: Linlin Liu, Qian Fu, Fei Hou, Ying He
- Abstract summary: We develop a new method for portrait image editing, which supports fine-grained editing of geometries, colors, lights and shadows using a single neural network model.
We adopt a novel asymmetric conditional GAN architecture: the generators take the transformed conditional inputs, such as edge maps, color palette, sliders and masks, that can be directly edited by the user.
We demonstrate the effectiveness of our method by evaluating it on the CelebAMask-HQ dataset with a wide range of tasks, including geometry/color/shadow/light editing, hand-drawn sketch to image translation, and color transfer.
- Score: 12.32304366243904
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We develop a new method for portrait image editing, which supports
fine-grained editing of geometries, colors, lights and shadows using a single
neural network model. We adopt a novel asymmetric conditional GAN architecture:
the generators take the transformed conditional inputs, such as edge maps,
color palette, sliders and masks, that can be directly edited by the user; the
discriminators take the conditional inputs in the way that can guide
controllable image generation more effectively. Taking color editing as an
example, we feed color palettes (which can be edited easily) into the
generator, and color maps (which contain positional information of colors) into
the discriminator. We also design a region-weighted discriminator so that
higher weights are assigned to more important regions, like eyes and skin.
Using a color palette, the user can directly specify the desired colors of
hair, skin, eyes, lip and background. Color sliders allow the user to blend
colors in an intuitive manner. The user can also edit lights and shadows by
modifying the corresponding masks. We demonstrate the effectiveness of our
method by evaluating it on the CelebAMask-HQ dataset with a wide range of
tasks, including geometry/color/shadow/light editing, hand-drawn sketch to
image translation, and color transfer. We also present ablation studies to
justify our design.
Related papers
- Automatic Controllable Colorization via Imagination [55.489416987587305]
We propose a framework for automatic colorization that allows for iterative editing and modifications.
By understanding the content within a grayscale image, we utilize a pre-trained image generation model to generate multiple images that contain the same content.
These images serve as references for coloring, mimicking the process of human experts.
arXiv Detail & Related papers (2024-04-08T16:46:07Z) - Mesh-Guided Neural Implicit Field Editing [42.78979161815414]
We propose a new approach that employs a mesh as a guiding mechanism in editing the neural field.
We first introduce a differentiable method using marching tetrahedra for polygonal mesh extraction from the neural implicit field.
We then design a differentiable color extractor to assign colors obtained from the volume renderings to this extracted mesh.
This differentiable colored mesh allows gradient back-propagation from the explicit mesh to the implicit fields, empowering users to easily manipulate the geometry and color of neural implicit fields.
arXiv Detail & Related papers (2023-12-04T18:59:58Z) - Dequantization and Color Transfer with Diffusion Models [5.228564799458042]
quantized images offer easy abstraction for patch-based edits and palette transfer.
We show that our model can generate natural images that respect the color palette the user asked for.
Our method can be usefully extended to another practical edit: recoloring patches of an image while respecting the source texture.
arXiv Detail & Related papers (2023-07-06T00:07:32Z) - RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color
Editing of 3D Scenes [21.284044381058575]
We present RecolorNeRF, a novel user-friendly color editing approach for neural radiance fields.
Our key idea is to decompose the scene into a set of pure-colored layers, forming a palette.
To support efficient palette-based editing, the color of each layer needs to be as representative as possible.
arXiv Detail & Related papers (2023-01-19T09:18:06Z) - PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields [60.66412075837952]
We present PaletteNeRF, a novel method for appearance editing of neural radiance fields (NeRF) based on 3D color decomposition.
Our method decomposes the appearance of each 3D point into a linear combination of palette-based bases.
We extend our framework with compressed semantic features for semantic-aware appearance editing.
arXiv Detail & Related papers (2022-12-21T00:20:01Z) - Hierarchical Vectorization for Portrait Images [12.32304366243904]
We propose a novel vectorization method that can automatically convert images into a 3-tier hierarchical representation.
The base layer consists of a set of sparse diffusion curves which characterize salient geometric features and low-frequency colors.
The middle level encodes specular highlights and shadows to large and editable Poisson regions (PR) and allows the user to directly adjust illumination.
The top level contains two types of pixel-sized PRs for high-frequency residuals and fine details such as pimples and pigmentation.
arXiv Detail & Related papers (2022-05-24T07:58:41Z) - HairCLIP: Design Your Hair by Text and Reference Image [100.85116679883724]
This paper proposes a new hair editing interaction mode, which enables manipulating hair attributes individually or jointly.
We encode the image and text conditions in a shared embedding space and propose a unified hair editing framework.
With the carefully designed network structures and loss functions, our framework can perform high-quality hair editing.
arXiv Detail & Related papers (2021-12-09T18:59:58Z) - SpaceEdit: Learning a Unified Editing Space for Open-Domain Image
Editing [94.31103255204933]
We propose a unified model for open-domain image editing focusing on color and tone adjustment of open-domain images.
Our model learns a unified editing space that is more semantic, intuitive, and easy to manipulate.
We show that by inverting image pairs into latent codes of the learned editing space, our model can be leveraged for various downstream editing tasks.
arXiv Detail & Related papers (2021-11-30T23:53:32Z) - SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches [95.45728042499836]
We propose a new paradigm of sketch-based image manipulation: mask-free local image manipulation.
Our model automatically predicts the target modification region and encodes it into a structure style vector.
A generator then synthesizes the new image content based on the style vector and sketch.
arXiv Detail & Related papers (2021-11-30T02:42:31Z) - Deep Saliency Prior for Reducing Visual Distraction [12.28561668097479]
We produce a range of powerful editing effects for reducing distraction in images.
The resulting effects are consistent with cognitive research on the human visual system.
We present results on a variety of natural images and conduct a perceptual study to evaluate and validate the changes in viewers' eye-gaze between the original images and our edited results.
arXiv Detail & Related papers (2021-09-05T03:19:21Z) - PIE: Portrait Image Embedding for Semantic Control [82.69061225574774]
We present the first approach for embedding real portrait images in the latent space of StyleGAN.
We use StyleRig, a pretrained neural network that maps the control space of a 3D morphable face model to the latent space of the GAN.
An identity energy preservation term allows spatially coherent edits while maintaining facial integrity.
arXiv Detail & Related papers (2020-09-20T17:53:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.