Towards Real-world Video Face Restoration: A New Benchmark
- URL: http://arxiv.org/abs/2404.19500v2
- Date: Sat, 4 May 2024 07:09:48 GMT
- Title: Towards Real-world Video Face Restoration: A New Benchmark
- Authors: Ziyan Chen, Jingwen He, Xinqi Lin, Yu Qiao, Chao Dong,
- Abstract summary: We introduce new real-world datasets named FOS with a taxonomy of "Full, Occluded, and Side" faces.
FOS datasets cover more diverse degradations and involve face samples from more complex scenarios.
We benchmarked both the state-of-the-art BFR methods and the video super resolution (VSR) methods to comprehensively study current approaches.
- Score: 33.01372704755186
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Blind face restoration (BFR) on images has significantly progressed over the last several years, while real-world video face restoration (VFR), which is more challenging for more complex face motions such as moving gaze directions and facial orientations involved, remains unsolved. Typical BFR methods are evaluated on privately synthesized datasets or self-collected real-world low-quality face images, which are limited in their coverage of real-world video frames. In this work, we introduced new real-world datasets named FOS with a taxonomy of "Full, Occluded, and Side" faces from mainly video frames to study the applicability of current methods on videos. Compared with existing test datasets, FOS datasets cover more diverse degradations and involve face samples from more complex scenarios, which helps to revisit current face restoration approaches more comprehensively. Given the established datasets, we benchmarked both the state-of-the-art BFR methods and the video super resolution (VSR) methods to comprehensively study current approaches, identifying their potential and limitations in VFR tasks. In addition, we studied the effectiveness of the commonly used image quality assessment (IQA) metrics and face IQA (FIQA) metrics by leveraging a subjective user study. With extensive experimental results and detailed analysis provided, we gained insights from the successes and failures of both current BFR and VSR methods. These results also pose challenges to current face restoration approaches, which we hope stimulate future advances in VFR research.
Related papers
- BFRFormer: Transformer-based generator for Real-World Blind Face
Restoration [37.77996097891398]
We propose a Transformer-based blind face restoration method, named BFRFormer, to reconstruct images with more identity-preserved details in an end-to-end manner.
Our method outperforms state-of-the-art methods on a synthetic dataset and four real-world datasets.
arXiv Detail & Related papers (2024-02-29T02:31:54Z) - Survey on Deep Face Restoration: From Non-blind to Blind and Beyond [79.1398990834247]
Face restoration (FR) is a specialized field within image restoration that aims to recover low-quality (LQ) face images into high-quality (HQ) face images.
Recent advances in deep learning technology have led to significant progress in FR methods.
arXiv Detail & Related papers (2023-09-27T08:39:03Z) - Enhancing Face Recognition with Latent Space Data Augmentation and
Facial Posture Reconstruction [8.493314424950599]
We propose an approach named the Face Representation Augmentation (FRA) for augmenting face datasets.
FRA is the first method that shifts its focus towards manipulating the face embeddings generated by any face representation learning algorithm.
The proposed method improves the identity classification accuracies by 9.52 %, 10.04 %, and 16.60 %, in comparison with the base models of MagFace, ArcFace, and CosFace, respectively.
arXiv Detail & Related papers (2023-01-27T20:54:58Z) - A Survey of Deep Face Restoration: Denoise, Super-Resolution, Deblur,
Artifact Removal [177.21001709272144]
Face Restoration (FR) aims to restore High-Quality (HQ) faces from Low-Quality (LQ) input images.
This paper comprehensively surveys recent advances in deep learning techniques for face restoration.
arXiv Detail & Related papers (2022-11-05T07:08:15Z) - Multi-Prior Learning via Neural Architecture Search for Blind Face
Restoration [61.27907052910136]
Blind Face Restoration (BFR) aims to recover high-quality face images from low-quality ones.
Current methods still suffer from two major difficulties: 1) how to derive a powerful network architecture without extensive hand tuning; 2) how to capture complementary information from multiple facial priors in one network to improve restoration performance.
We propose a Face Restoration Searching Network (FRSNet) to adaptively search the suitable feature extraction architecture within our specified search space.
arXiv Detail & Related papers (2022-06-28T12:29:53Z) - Blind Face Restoration: Benchmark Datasets and a Baseline Model [63.053331687284064]
Blind Face Restoration (BFR) aims to construct a high-quality (HQ) face image from its corresponding low-quality (LQ) input.
We first synthesize two blind face restoration benchmark datasets called EDFace-Celeb-1M (BFR128) and EDFace-Celeb-150K (BFR512)
State-of-the-art methods are benchmarked on them under five settings including blur, noise, low resolution, JPEG compression artifacts, and the combination of them (full degradation)
arXiv Detail & Related papers (2022-06-08T06:34:24Z) - Joint Face Image Restoration and Frontalization for Recognition [79.78729632975744]
In real-world scenarios, many factors may harm face recognition performance, e.g., large pose, bad illumination,low resolution, blur and noise.
Previous efforts usually first restore the low-quality faces to high-quality ones and then perform face recognition.
We propose an Multi-Degradation Face Restoration model to restore frontalized high-quality faces from the given low-quality ones.
arXiv Detail & Related papers (2021-05-12T03:52:41Z) - HiFaceGAN: Face Renovation via Collaborative Suppression and
Replenishment [63.333407973913374]
"Face Renovation"(FR) is a semantic-guided generation problem.
"HiFaceGAN" is a multi-stage framework containing several nested CSR units.
experiments on both synthetic and real face images have verified the superior performance of HiFaceGAN.
arXiv Detail & Related papers (2020-05-11T11:33:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.