ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration

• URL: http://arxiv.org/abs/2412.05043v1
• Date: Fri, 06 Dec 2024 13:49:10 GMT
• Title: ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration
• Authors: Chi-Wei Hsiao, Yu-Lun Liu, Cheng-Kun Yang, Sheng-Po Kuo, Kevin Jou, Chia-Ping Chen,
• Abstract summary: We propose ReF-LDM, an adaptation of LDM designed to generate HQ face images conditioned on one LQ image and multiple HQ reference images. Our model integrates an effective and efficient mechanism, CacheKV, to leverage the reference images during the generation process. Lastly, we construct FFHQ-Ref, a dataset consisting of 20,405 high-quality (HQ) face images with corresponding reference images.
• Score: 11.712490684089609
• License:
• Abstract: While recent works on blind face image restoration have successfully produced impressive high-quality (HQ) images with abundant details from low-quality (LQ) input images, the generated content may not accurately reflect the real appearance of a person. To address this problem, incorporating well-shot personal images as additional reference inputs could be a promising strategy. Inspired by the recent success of the Latent Diffusion Model (LDM), we propose ReF-LDM, an adaptation of LDM designed to generate HQ face images conditioned on one LQ image and multiple HQ reference images. Our model integrates an effective and efficient mechanism, CacheKV, to leverage the reference images during the generation process. Additionally, we design a timestep-scaled identity loss, enabling our LDM-based model to focus on learning the discriminating features of human faces. Lastly, we construct FFHQ-Ref, a dataset consisting of 20,405 high-quality (HQ) face images with corresponding reference images, which can serve as both training and evaluation data for reference-based face restoration models.

Related papers

• OSDFace: One-Step Diffusion Model for Face Restoration [72.5045389847792]
  Diffusion models have demonstrated impressive performance in face restoration. We propose OSDFace, a novel one-step diffusion model for face restoration. Results demonstrate that OSDFace surpasses current state-of-the-art (SOTA) methods in both visual quality and quantitative metrics.
  arXiv  Detail & Related papers  (2024-11-26T07:07:48Z)
• Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model [55.46927355649013]
  We introduce a novel Multi-modal Guided Real-World Face Restoration technique. MGFR can mitigate the generation of false facial attributes and identities. We present the Reface-HQ dataset, comprising over 23,000 high-resolution facial images across 5,000 identities.
  arXiv  Detail & Related papers  (2024-10-05T13:46:56Z)
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation [52.509092010267665]
  We introduce LlamaGen, a new family of image generation models that apply original next-token prediction'' paradigm of large language models to visual generation domain. It is an affirmative answer to whether vanilla autoregressive models, e.g., Llama, without inductive biases on visual signals can achieve state-of-the-art image generation performance if scaling properly.
  arXiv  Detail & Related papers  (2024-06-10T17:59:52Z)
• DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception [66.88792390480343]
  We propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data, and a smaller base model size.
  arXiv  Detail & Related papers  (2024-05-24T05:46:04Z)
• PFStorer: Personalized Face Restoration and Super-Resolution [19.479263766534345]
  Recent developments in face restoration have achieved remarkable results in producing high-quality and lifelike outputs. The stunning results however often fail to be faithful with respect to the identity of the person as the models lack necessary context. In our approach a restoration model is personalized using a few images of the identity, leading to tailored restoration with respect to the identity while retaining fine-grained details.
  arXiv  Detail & Related papers  (2024-03-13T11:39:30Z)
• Learning from History: Task-agnostic Model Contrastive Learning for Image Restoration [79.04007257606862]
  This paper introduces an innovative method termed 'learning from history', which dynamically generates negative samples from the target model itself. Our approach, named Model Contrastive Learning for Image Restoration (MCLIR), rejuvenates latency models as negative models, making it compatible with diverse image restoration tasks.
  arXiv  Detail & Related papers  (2023-09-12T07:50:54Z)
• Synthesizing Realistic Image Restoration Training Pairs: A Diffusion Approach [24.41545801521035]
  In supervised image restoration tasks, one key issue is how to obtain the aligned high-quality (HQ) and low-quality (LQ) training image pairs. We propose a new approach to synthesizing realistic image restoration training pairs using the emerging denoising diffusion probabilistic model (DDPM) Thanks to the strong capability of DDPM in distribution approximation, the synthesized HQ-LQ image pairs can be used to train robust models for real-world image restoration tasks.
  arXiv  Detail & Related papers  (2023-03-13T10:49:59Z)
• Super-resolution Reconstruction of Single Image for Latent features [8.857209365343646]
  Single-image super-resolution (SISR) typically focuses on restoring various degraded low-resolution (LR) images to a single high-resolution (HR) image. It is often challenging for models to simultaneously maintain high quality and rapid sampling while preserving diversity in details and texture features. This challenge can lead to issues such as model collapse, lack of rich details and texture features in the reconstructed HR images, and excessive time consumption for model sampling.
  arXiv  Detail & Related papers  (2022-11-16T09:37:07Z)
• Joint Face Image Restoration and Frontalization for Recognition [79.78729632975744]
  In real-world scenarios, many factors may harm face recognition performance, e.g., large pose, bad illumination,low resolution, blur and noise. Previous efforts usually first restore the low-quality faces to high-quality ones and then perform face recognition. We propose an Multi-Degradation Face Restoration model to restore frontalized high-quality faces from the given low-quality ones.
  arXiv  Detail & Related papers  (2021-05-12T03:52:41Z)
• Joint Face Completion and Super-resolution using Multi-scale Feature Relation Learning [26.682678558621625]
  This paper proposes a multi-scale feature graph generative adversarial network (MFG-GAN) to implement the face restoration of images in which both degradation modes coexist. Based on the GAN, the MFG-GAN integrates the graph convolution and feature pyramid network to restore occluded low-resolution face images to non-occluded high-resolution face images. Experimental results on the public-domain CelebA and Helen databases show that the proposed approach outperforms state-of-the-art methods in performing face super-resolution (up to 4x or 8x) and face completion simultaneously.
  arXiv  Detail & Related papers  (2020-02-29T13:31:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.