A Deep Learning Framework to Reconstruct Face under Mask

• URL: http://arxiv.org/abs/2203.12482v1
• Date: Wed, 23 Mar 2022 15:23:24 GMT
• Title: A Deep Learning Framework to Reconstruct Face under Mask
• Authors: Gourango Modak, Shuvra Smaran Das, Md. Ajharul Islam Miraj, Md. Kishor Morol
• Abstract summary: The purpose of this work is to extract the mask region from a masked image and rebuild the area that has been detected. This problem is complex because (i) it is difficult to determine the gender of an image hidden behind a mask, which causes the network to become confused and reconstruct the male face as a female or vice versa. To solve this complex task, we split the problem into three phases: landmark detection, object detection for the targeted mask area, and inpainting the addressed mask region.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While deep learning-based image reconstruction methods have shown significant success in removing objects from pictures, they have yet to achieve acceptable results for attributing consistency to gender, ethnicity, expression, and other characteristics like the topological structure of the face. The purpose of this work is to extract the mask region from a masked image and rebuild the area that has been detected. This problem is complex because (i) it is difficult to determine the gender of an image hidden behind a mask, which causes the network to become confused and reconstruct the male face as a female or vice versa; (ii) we may receive images from multiple angles, making it extremely difficult to maintain the actual shape, topological structure of the face and a natural image; and (iii) there are problems with various mask forms because, in some cases, the area of the mask cannot be anticipated precisely; certain parts of the mask remain on the face after completion. To solve this complex task, we split the problem into three phases: landmark detection, object detection for the targeted mask area, and inpainting the addressed mask region. To begin, to solve the first problem, we have used gender classification, which detects the actual gender behind a mask, then we detect the landmark of the masked facial image. Second, we identified the non-face item, i.e., the mask, and used the Mask R-CNN network to create the binary mask of the observed mask area. Thirdly, we developed an inpainting network that uses anticipated landmarks to create realistic images. To segment the mask, this article uses a mask R-CNN and offers a binary segmentation map for identifying the mask area. Additionally, we generated the image utilizing landmarks as structural guidance through a GAN-based network. The studies presented in this paper use the FFHQ and CelebA datasets.

Related papers

• Face Mask Removal with Region-attentive Face Inpainting [0.7433327915285965]
  We propose a generative face inpainting method to recover/reconstruct the masked part of a face. Our proposed method includes a Multi-scale Channel-Spatial Attention Module (M-CSAM) to mitigate the spatial information loss. We synthesize our own Masked-Faces dataset from the CelebA dataset by incorporating five different types of face masks.
  arXiv  Detail & Related papers  (2024-09-10T20:10:11Z)
• Seeing through the Mask: Multi-task Generative Mask Decoupling Face Recognition [47.248075664420874]
  Current general face recognition system suffers from serious performance degradation when encountering occluded scenes. This paper proposes a Multi-task gEnerative mask dEcoupling face Recognition (MEER) network to jointly handle these two tasks. We first present a novel mask decoupling module to disentangle mask and identity information, which makes the network obtain purer identity features from visible facial components.
  arXiv  Detail & Related papers  (2023-11-20T03:23:03Z)
• Shape-Aware Masking for Inpainting in Medical Imaging [49.61617087640379]
  Inpainting has been proposed as a successful deep learning technique for unsupervised medical image model discovery. We introduce a method for generating shape-aware masks for inpainting, which aims at learning the statistical shape prior. We propose an unsupervised guided masking approach based on an off-the-shelf inpainting model and a superpixel over-segmentation algorithm.
  arXiv  Detail & Related papers  (2022-07-12T18:35:17Z)
• Layered Depth Refinement with Mask Guidance [61.10654666344419]
  We formulate a novel problem of mask-guided depth refinement that utilizes a generic mask to refine the depth prediction of SIDE models. Our framework performs layered refinement and inpainting/outpainting, decomposing the depth map into two separate layers signified by the mask and the inverse mask. We empirically show that our method is robust to different types of masks and initial depth predictions, accurately refining depth values in inner and outer mask boundary regions.
  arXiv  Detail & Related papers  (2022-06-07T06:42:44Z)
• A Unified Framework for Masked and Mask-Free Face Recognition via Feature Rectification [19.417191498842044]
  We propose a unified framework, named Face Feature Rectification Network (FFR-Net), for recognizing both masked and mask-free faces alike. We introduce rectification blocks to rectify features extracted by a state-of-the-art recognition model, in both spatial and channel dimensions. Experiments show that our framework can learn a rectified feature space for recognizing both masked and mask-free faces effectively.
  arXiv  Detail & Related papers  (2022-02-15T12:37:59Z)
• MLFW: A Database for Face Recognition on Masked Faces [56.441078419992046]
  Masked LFW (MLFW) is a tool to generate masked faces from unmasked faces automatically. The recognition accuracy of SOTA models declines 5%-16% on MLFW database compared with the accuracy on the original images.
  arXiv  Detail & Related papers  (2021-09-13T09:30:10Z)
• Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness [66.55719330810547]
  Inpainting arbitrary missing regions is challenging because learning valid features for various masked regions is nontrivial. We propose a novel mask-aware inpainting solution that learns multi-scale features for missing regions in the encoding phase. Our framework is validated both quantitatively and qualitatively via extensive experiments on three public datasets.
  arXiv  Detail & Related papers  (2021-04-28T13:17:47Z)
• Towards NIR-VIS Masked Face Recognition [47.00916333095693]
  Near-infrared to visible (NIR-VIS) face recognition is the most common case in heterogeneous face recognition. We propose a novel training method to maximize the mutual information shared by the face representation of two domains. In addition, a 3D face reconstruction based approach is employed to synthesize masked face from the existing NIR image.
  arXiv  Detail & Related papers  (2021-04-14T10:40:09Z)
• Deep Learning Framework to Detect Face Masks from Video Footage [0.0]
  We propose an approach for detecting facial masks in videos using deep learning. The proposed framework capitalizes on the MTCNN face detection model to identify the faces and their corresponding facial landmarks present in the video frame. The proposed methodology demonstrated its effectiveness in detecting facial masks by achieving high precision, recall, and accuracy.
  arXiv  Detail & Related papers  (2020-11-04T16:02:03Z)
• MaskedFace-Net -- A Dataset of Correctly/Incorrectly Masked Face Images in the Context of COVID-19 [2.7528170226206443]
  The wearing of the face masks appears as a solution for limiting the spread of COVID-19. To perform this task, a large dataset of masked faces is necessary for training deep learning models towards detecting people wearing masks and those not wearing masks. Some large datasets of masked faces are available in the literature. However, at the moment, there are no available large dataset of masked face images that permits to check if detected masked faces are correctly worn or not.
  arXiv  Detail & Related papers  (2020-08-18T16:38:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.