When Diffusion Models Forget Who You Are: Identity Preservation in Face Inpainting under Large Occlusions

• URL: http://arxiv.org/abs/2608.04820v1
• Date: Wed, 05 Aug 2026 13:23:40 GMT
• Title: When Diffusion Models Forget Who You Are: Identity Preservation in Face Inpainting under Large Occlusions
• Abstract summary: ReSem-Face is a cascaded diffusion framework that introduces an explicit identity-conditioned semantic prior for face inpainting.<n>Our approach distills representative identity features from multiple references to reconstruct missing semantic regions.<n> Experiments on CelebAHQ-IDI-5 and VGGFace2 demonstrate that ReSem-Face yields more reliable identity-preserving completion under severe semantic masks.
• Score: 14.50832575272006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Face inpainting with diffusion models has recently achieved impressive visual quality, yet preserving identity fidelity under significant occlusion and conflicting text guidance remains a major challenge. To address this issue, we present Reference Semantic Inpainting for Face (ReSem-Face), a cascaded diffusion framework that introduces an explicit identity-conditioned semantic prior for multi-reference face inpainting. Our approach distills representative identity features from multiple references to reconstruct missing semantic regions, which then guide the diffusion process through a multi-stream conditioning architecture. This design provides strong semantic constraints when pixels are absent and stabilizes identity reconstruction while remaining compatible with prompt-driven edits. Experiments on CelebAHQ-IDI-5 and VGGFace2 demonstrate that ReSem-Face yields more reliable identity-preserving completion under severe semantic masks and improves text-controlled editing quality compared with representative baselines.

Related papers

• Face inpainting with Identity Preserving Latent Diffusion Models [10.348748950921667]
  ID-ControlNet is an identity-preserving face inpainting framework built upon latent diffusion models.<n>We introduce an identity consistency and triplet loss training strategy that explicitly enforces alignment between the generated face and the target identity representation.
  arXiv  Detail & Related papers  (2026-05-15T23:19:43Z)
• Optimizing ID Consistency in Multimodal Large Models: Facial Restoration via Alignment, Entanglement, and Disentanglement [54.199726425201895]
  Multimodal editing large models have demonstrated powerful editing capabilities across diverse tasks.<n>Current facial ID preservation methods struggle to achieve consistent restoration of both facial identity and edited element IP.<n>We propose EditedID, an Alignment-Disentanglement-Entanglement framework for robust identity-specific facial restoration.
  arXiv  Detail & Related papers  (2026-02-21T08:24:42Z)
• SIDeR: Semantic Identity Decoupling for Unrestricted Face Privacy [53.75084833636302]
  We propose SIDeR, a Semantic decoupling-driven framework for unrestricted face privacy protection.<n> SIDeR decomposes a facial image into a machine-recognizable identity feature vector and a visually perceptible semantic appearance component.<n>For authorized access, SIDeR can be restored to its original form when the correct password is provided.
  arXiv  Detail & Related papers  (2026-02-04T19:30:48Z)
• BeyondFacial: Identity-Preserving Personalized Generation Beyond Facial Close-ups [22.017690133402912]
  Identity-Preserving Personalized Generation (I) has advanced film production and artistic creation, yet existing approaches overemphasize facial regions.<n>These methods suffer from weak visual narrativity and poor semantic consistency under complex text prompts.<n>This paper presents an I method that breaks the constraint of facial close-ups, achieving synergistic optimization of identity fidelity and scene semantic creation.
  arXiv  Detail & Related papers  (2025-11-15T01:56:14Z)
• Robust ID-Specific Face Restoration via Alignment Learning [14.7430941613282]
  We present Robust ID-Specific Face Restoration (RIDFR), a novel ID-specific face restoration framework based on diffusion models.<n>RIDFR incorporates Alignment Learning, which aligns the restoration results from multiple references with the same identity in order to suppress the interference of ID-irrelevant face semantics.<n>Experiments demonstrate that our framework outperforms the state-of-the-art methods, reconstructing high-quality ID-specific results with high identity fidelity and demonstrating strong robustness.
  arXiv  Detail & Related papers  (2025-07-15T03:16:12Z)
• OSDFace: One-Step Diffusion Model for Face Restoration [72.5045389847792]
  Diffusion models have demonstrated impressive performance in face restoration.<n>We propose OSDFace, a novel one-step diffusion model for face restoration.<n>Results demonstrate that OSDFace surpasses current state-of-the-art (SOTA) methods in both visual quality and quantitative metrics.
  arXiv  Detail & Related papers  (2024-11-26T07:07:48Z)
• CLR-Face: Conditional Latent Refinement for Blind Face Restoration Using Score-Based Diffusion Models [57.9771859175664]
  Recent generative-prior-based methods have shown promising blind face restoration performance. Generating fine-grained facial details faithful to inputs remains a challenging problem. We introduce a diffusion-based-prior inside a VQGAN architecture that focuses on learning the distribution over uncorrupted latent embeddings.
  arXiv  Detail & Related papers  (2024-02-08T23:51:49Z)
• Personalized Face Inpainting with Diffusion Models by Parallel Visual Attention [55.33017432880408]
  This paper proposes the use of Parallel Visual Attention (PVA) in conjunction with diffusion models to improve inpainting results. We train the added attention modules and identity encoder on CelebAHQ-IDI, a dataset proposed for identity-preserving face inpainting. Experiments demonstrate that PVA attains unparalleled identity resemblance in both face inpainting and face inpainting with language guidance tasks.
  arXiv  Detail & Related papers  (2023-12-06T15:39:03Z)
• When StyleGAN Meets Stable Diffusion: a $\mathscr{W}_+$ Adapter for Personalized Image Generation [60.305112612629465]
  Text-to-image diffusion models have excelled in producing diverse, high-quality, and photo-realistic images. We present a novel use of the extended StyleGAN embedding space $mathcalW_+$ to achieve enhanced identity preservation and disentanglement for diffusion models. Our method adeptly generates personalized text-to-image outputs that are not only compatible with prompt descriptions but also amenable to common StyleGAN editing directions.
  arXiv  Detail & Related papers  (2023-11-29T09:05:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.