Towards Authentic Face Restoration with Iterative Diffusion Models and Beyond

• URL: http://arxiv.org/abs/2307.08996v1
• Date: Tue, 18 Jul 2023 06:31:01 GMT
• Title: Towards Authentic Face Restoration with Iterative Diffusion Models and Beyond
• Authors: Yang Zhao, Tingbo Hou, Yu-Chuan Su, Xuhui Jia. Yandong Li and Matthias Grundmann
• Abstract summary: We propose $textbfIDM$, an $textbfI$teratively learned face restoration system based on denoising $textbfD$iffusion. We demonstrate superior performance on blind face restoration tasks.
• Score: 30.114913184727
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: An authentic face restoration system is becoming increasingly demanding in many computer vision applications, e.g., image enhancement, video communication, and taking portrait. Most of the advanced face restoration models can recover high-quality faces from low-quality ones but usually fail to faithfully generate realistic and high-frequency details that are favored by users. To achieve authentic restoration, we propose $\textbf{IDM}$, an $\textbf{I}$teratively learned face restoration system based on denoising $\textbf{D}$iffusion $\textbf{M}$odels (DDMs). We define the criterion of an authentic face restoration system, and argue that denoising diffusion models are naturally endowed with this property from two aspects: intrinsic iterative refinement and extrinsic iterative enhancement. Intrinsic learning can preserve the content well and gradually refine the high-quality details, while extrinsic enhancement helps clean the data and improve the restoration task one step further. We demonstrate superior performance on blind face restoration tasks. Beyond restoration, we find the authentically cleaned data by the proposed restoration system is also helpful to image generation tasks in terms of training stabilization and sample quality. Without modifying the models, we achieve better quality than state-of-the-art on FFHQ and ImageNet generation using either GANs or diffusion models.

Related papers

• AuthFace: Towards Authentic Blind Face Restoration with Face-oriented Generative Diffusion Prior [13.27748226506837]
  Blind face restoration (BFR) is a fundamental and challenging problem in computer vision. Recent research endeavors rely on facial image priors from the powerful pretrained text-to-image (T2I) diffusion models. We propose AuthFace, which achieves highly authentic face restoration results by exploring a face-oriented generative diffusion prior.
  arXiv  Detail & Related papers  (2024-10-13T14:56:13Z)
• Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model [55.46927355649013]
  We introduce a novel Multi-modal Guided Real-World Face Restoration technique. MGFR can mitigate the generation of false facial attributes and identities. We present the Reface-HQ dataset, comprising over 23,000 high-resolution facial images across 5,000 identities.
  arXiv  Detail & Related papers  (2024-10-05T13:46:56Z)
• Diff-Restorer: Unleashing Visual Prompts for Diffusion-based Universal Image Restoration [19.87693298262894]
  We propose Diff-Restorer, a universal image restoration method based on the diffusion model. We utilize the pre-trained visual language model to extract visual prompts from degraded images. We also design a Degradation-aware Decoder to perform structural correction and convert the latent code to the pixel domain.
  arXiv  Detail & Related papers  (2024-07-04T05:01:10Z)
• PFStorer: Personalized Face Restoration and Super-Resolution [19.479263766534345]
  Recent developments in face restoration have achieved remarkable results in producing high-quality and lifelike outputs. The stunning results however often fail to be faithful with respect to the identity of the person as the models lack necessary context. In our approach a restoration model is personalized using a few images of the identity, leading to tailored restoration with respect to the identity while retaining fine-grained details.
  arXiv  Detail & Related papers  (2024-03-13T11:39:30Z)
• CLR-Face: Conditional Latent Refinement for Blind Face Restoration Using Score-Based Diffusion Models [57.9771859175664]
  Recent generative-prior-based methods have shown promising blind face restoration performance. Generating fine-grained facial details faithful to inputs remains a challenging problem. We introduce a diffusion-based-prior inside a VQGAN architecture that focuses on learning the distribution over uncorrupted latent embeddings.
  arXiv  Detail & Related papers  (2024-02-08T23:51:49Z)
• InstructIR: High-Quality Image Restoration Following Human Instructions [61.1546287323136]
  We present the first approach that uses human-written instructions to guide the image restoration model. Our method, InstructIR, achieves state-of-the-art results on several restoration tasks.
  arXiv  Detail & Related papers  (2024-01-29T18:53:33Z)
• Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild [57.06779516541574]
  SUPIR (Scaling-UP Image Restoration) is a groundbreaking image restoration method that harnesses generative prior and the power of model scaling up. We collect a dataset comprising 20 million high-resolution, high-quality images for model training, each enriched with descriptive text annotations.
  arXiv  Detail & Related papers  (2024-01-24T17:58:07Z)
• Towards Real-World Blind Face Restoration with Generative Diffusion Prior [69.84480964328465]
  Blind face restoration is an important task in computer vision and has gained significant attention due to its wide-range applications. We propose BFRffusion which is thoughtfully designed to effectively extract features from low-quality face images. We also build a privacy-preserving face dataset called PFHQ with balanced attributes like race, gender, and age.
  arXiv  Detail & Related papers  (2023-12-25T14:16:24Z)
• RestoreFormer++: Towards Real-World Blind Face Restoration from Undegraded Key-Value Pairs [63.991802204929485]
  Blind face restoration aims at recovering high-quality face images from those with unknown degradations. Current algorithms mainly introduce priors to complement high-quality details and achieve impressive progress. We propose RestoreFormer++, which introduces fully-spatial attention mechanisms to model the contextual information and the interplay with the priors. We show that RestoreFormer++ outperforms state-of-the-art algorithms on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-08-14T16:04:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.