Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model

• URL: http://arxiv.org/abs/2410.04161v1
• Date: Sat, 5 Oct 2024 13:46:56 GMT
• Title: Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model
• Authors: Keda Tao, Jinjin Gu, Yulun Zhang, Xiucheng Wang, Nan Cheng,
• Abstract summary: We introduce a novel Multi-modal Guided Real-World Face Restoration technique. MGFR can mitigate the generation of false facial attributes and identities. We present the Reface-HQ dataset, comprising over 23,000 high-resolution facial images across 5,000 identities.
• Score: 55.46927355649013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a novel Multi-modal Guided Real-World Face Restoration (MGFR) technique designed to improve the quality of facial image restoration from low-quality inputs. Leveraging a blend of attribute text prompts, high-quality reference images, and identity information, MGFR can mitigate the generation of false facial attributes and identities often associated with generative face restoration methods. By incorporating a dual-control adapter and a two-stage training strategy, our method effectively utilizes multi-modal prior information for targeted restoration tasks. We also present the Reface-HQ dataset, comprising over 23,000 high-resolution facial images across 5,000 identities, to address the need for reference face training images. Our approach achieves superior visual quality in restoring facial details under severe degradation and allows for controlled restoration processes, enhancing the accuracy of identity preservation and attribute correction. Including negative quality samples and attribute prompts in the training further refines the model's ability to generate detailed and perceptually accurate images.

Related papers

• AuthFace: Towards Authentic Blind Face Restoration with Face-oriented Generative Diffusion Prior [13.27748226506837]
  Blind face restoration (BFR) is a fundamental and challenging problem in computer vision. Recent research endeavors rely on facial image priors from the powerful pretrained text-to-image (T2I) diffusion models. We propose AuthFace, which achieves highly authentic face restoration results by exploring a face-oriented generative diffusion prior.
  arXiv  Detail & Related papers  (2024-10-13T14:56:13Z)
• DSL-FIQA: Assessing Facial Image Quality via Dual-Set Degradation Learning and Landmark-Guided Transformer [23.70791030264281]
  Generic Face Image Quality Assessment (GFIQA) evaluates the perceptual quality of facial images. We present a novel transformer-based method for GFIQA, which is aided by two unique mechanisms.
  arXiv  Detail & Related papers  (2024-06-13T23:11:25Z)
• PFStorer: Personalized Face Restoration and Super-Resolution [19.479263766534345]
  Recent developments in face restoration have achieved remarkable results in producing high-quality and lifelike outputs. The stunning results however often fail to be faithful with respect to the identity of the person as the models lack necessary context. In our approach a restoration model is personalized using a few images of the identity, leading to tailored restoration with respect to the identity while retaining fine-grained details.
  arXiv  Detail & Related papers  (2024-03-13T11:39:30Z)
• Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild [57.06779516541574]
  SUPIR (Scaling-UP Image Restoration) is a groundbreaking image restoration method that harnesses generative prior and the power of model scaling up. We collect a dataset comprising 20 million high-resolution, high-quality images for model training, each enriched with descriptive text annotations.
  arXiv  Detail & Related papers  (2024-01-24T17:58:07Z)
• RestoreFormer++: Towards Real-World Blind Face Restoration from Undegraded Key-Value Pairs [63.991802204929485]
  Blind face restoration aims at recovering high-quality face images from those with unknown degradations. Current algorithms mainly introduce priors to complement high-quality details and achieve impressive progress. We propose RestoreFormer++, which introduces fully-spatial attention mechanisms to model the contextual information and the interplay with the priors. We show that RestoreFormer++ outperforms state-of-the-art algorithms on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-08-14T16:04:53Z)
• Towards Authentic Face Restoration with Iterative Diffusion Models and Beyond [30.114913184727]
  We propose $textbfIDM$, an $textbfI$teratively learned face restoration system based on denoising $textbfD$iffusion. We demonstrate superior performance on blind face restoration tasks.
  arXiv  Detail & Related papers  (2023-07-18T06:31:01Z)
• Multi-Prior Learning via Neural Architecture Search for Blind Face Restoration [61.27907052910136]
  Blind Face Restoration (BFR) aims to recover high-quality face images from low-quality ones. Current methods still suffer from two major difficulties: 1) how to derive a powerful network architecture without extensive hand tuning; 2) how to capture complementary information from multiple facial priors in one network to improve restoration performance. We propose a Face Restoration Searching Network (FRSNet) to adaptively search the suitable feature extraction architecture within our specified search space.
  arXiv  Detail & Related papers  (2022-06-28T12:29:53Z)
• RestoreFormer: High-Quality Blind Face Restoration From Undegraded Key-Value Pairs [48.33214614798882]
  We propose RestoreFormer, which explores fully-spatial attentions to model contextual information. It learns fully-spatial interactions between corrupted queries and high-quality key-value pairs. It outperforms advanced state-of-the-art methods on one synthetic dataset and three real-world datasets.
  arXiv  Detail & Related papers  (2022-01-17T12:21:55Z)
• Network Architecture Search for Face Enhancement [82.25775020564654]
  We present a multi-task face restoration network, called Network Architecture Search for Face Enhancement (NASFE) NASFE can enhance poor quality face images containing a single degradation (i.e. noise or blur) or multiple degradations (noise+blur+low-light)
  arXiv  Detail & Related papers  (2021-05-13T19:46:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.