PFStorer: Personalized Face Restoration and Super-Resolution
- URL: http://arxiv.org/abs/2403.08436v1
- Date: Wed, 13 Mar 2024 11:39:30 GMT
- Title: PFStorer: Personalized Face Restoration and Super-Resolution
- Authors: Tuomas Varanka, Tapani Toivonen, Soumya Tripathy, Guoying Zhao, Erman
Acar
- Abstract summary: Recent developments in face restoration have achieved remarkable results in producing high-quality and lifelike outputs.
The stunning results however often fail to be faithful with respect to the identity of the person as the models lack necessary context.
In our approach a restoration model is personalized using a few images of the identity, leading to tailored restoration with respect to the identity while retaining fine-grained details.
- Score: 19.479263766534345
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recent developments in face restoration have achieved remarkable results in
producing high-quality and lifelike outputs. The stunning results however often
fail to be faithful with respect to the identity of the person as the models
lack necessary context. In this paper, we explore the potential of personalized
face restoration with diffusion models. In our approach a restoration model is
personalized using a few images of the identity, leading to tailored
restoration with respect to the identity while retaining fine-grained details.
By using independent trainable blocks for personalization, the rich prior of a
base restoration model can be exploited to its fullest. To avoid the model
relying on parts of identity left in the conditioning low-quality images, a
generative regularizer is employed. With a learnable parameter, the model
learns to balance between the details generated based on the input image and
the degree of personalization. Moreover, we improve the training pipeline of
face restoration models to enable an alignment-free approach. We showcase the
robust capabilities of our approach in several real-world scenarios with
multiple identities, demonstrating our method's ability to generate
fine-grained details with faithful restoration. In the user study we evaluate
the perceptual quality and faithfulness of the genereated details, with our
method being voted best 61% of the time compared to the second best with 25% of
the votes.
Related papers
- Personalized Restoration via Dual-Pivot Tuning [18.912158172904654]
We propose a simple, yet effective, method for personalized restoration, called Dual-Pivot Tuning.
Our key observation is that for optimal personalization, the generative model should be tuned around a fixed text pivot.
This approach ensures that personalization does not interfere with the restoration process, resulting in a natural appearance with high fidelity to the person's identity and the attributes of the degraded image.
arXiv Detail & Related papers (2023-12-28T18:57:49Z) - Restoration by Generation with Constrained Priors [25.906981634736795]
We propose a method to adapt a pretrained diffusion model for image restoration by simply adding noise to the input image to be restored and then denoise.
We show superior performances on multiple real-world restoration datasets in preserving identity and image quality.
This approach allows us to produce results that accurately preserve high-frequency details, which previous works are unable to do.
arXiv Detail & Related papers (2023-12-28T17:50:54Z) - Effective Adapter for Face Recognition in the Wild [72.75516495170199]
We tackle the challenge of face recognition in the wild, where images often suffer from low quality and real-world distortions.
Traditional approaches-either training models directly on degraded images or their enhanced counterparts using face restoration techniques-have proven ineffective.
We propose an effective adapter for augmenting existing face recognition models trained on high-quality facial datasets.
arXiv Detail & Related papers (2023-12-04T08:55:46Z) - RestoreFormer++: Towards Real-World Blind Face Restoration from
Undegraded Key-Value Pairs [63.991802204929485]
Blind face restoration aims at recovering high-quality face images from those with unknown degradations.
Current algorithms mainly introduce priors to complement high-quality details and achieve impressive progress.
We propose RestoreFormer++, which introduces fully-spatial attention mechanisms to model the contextual information and the interplay with the priors.
We show that RestoreFormer++ outperforms state-of-the-art algorithms on both synthetic and real-world datasets.
arXiv Detail & Related papers (2023-08-14T16:04:53Z) - Towards Authentic Face Restoration with Iterative Diffusion Models and
Beyond [30.114913184727]
We propose $textbfIDM$, an $textbfI$teratively learned face restoration system based on denoising $textbfD$iffusion.
We demonstrate superior performance on blind face restoration tasks.
arXiv Detail & Related papers (2023-07-18T06:31:01Z) - Identity Encoder for Personalized Diffusion [57.1198884486401]
We propose an encoder-based approach for personalization.
We learn an identity encoder which can extract an identity representation from a set of reference images of a subject.
We show that our approach consistently outperforms existing fine-tuning based approach in both image generation and reconstruction.
arXiv Detail & Related papers (2023-04-14T23:32:24Z) - Attribute-preserving Face Dataset Anonymization via Latent Code
Optimization [64.4569739006591]
We present a task-agnostic anonymization procedure that directly optimize the images' latent representation in the latent space of a pre-trained GAN.
We demonstrate through a series of experiments that our method is capable of anonymizing the identity of the images whilst -- crucially -- better-preserving the facial attributes.
arXiv Detail & Related papers (2023-03-20T17:34:05Z) - Learning Dual Memory Dictionaries for Blind Face Restoration [75.66195723349512]
Recent works mainly treat the two aspects, i.e., generic and specific restoration, separately.
This paper suggests a DMDNet by explicitly memorizing the generic and specific features through dual dictionaries.
A new high-quality dataset, termed CelebRef-HQ, is constructed to promote the exploration of specific face restoration in the high-resolution space.
arXiv Detail & Related papers (2022-10-15T01:55:41Z) - MyStyle: A Personalized Generative Prior [38.3436972491162]
We introduce MyStyle, a personalized deep generative prior trained with a few shots of an individual.
MyStyle allows to reconstruct, enhance and edit images of a specific person.
arXiv Detail & Related papers (2022-03-31T17:59:19Z) - RestoreFormer: High-Quality Blind Face Restoration From Undegraded
Key-Value Pairs [48.33214614798882]
We propose RestoreFormer, which explores fully-spatial attentions to model contextual information.
It learns fully-spatial interactions between corrupted queries and high-quality key-value pairs.
It outperforms advanced state-of-the-art methods on one synthetic dataset and three real-world datasets.
arXiv Detail & Related papers (2022-01-17T12:21:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.