IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off

• URL: http://arxiv.org/abs/2604.00903v1
• Date: Wed, 01 Apr 2026 13:46:22 GMT
• Title: IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off
• Abstract summary: We propose a model-side defense that integrates identity decoupling into the personalization pipeline.<n>IDDM consistently reduces identity linkability while preserving high-quality personalized generation.
• Score: 16.978864881456392
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized text-to-image diffusion models (e.g., DreamBooth, LoRA) enable users to synthesize high-fidelity avatars from a few reference photos for social expression. However, once these generations are shared on social media platforms (e.g., Instagram, Facebook), they can be linked to the real user via face recognition systems, enabling identity tracking and profiling. Existing defenses mainly follow an anti-personalization strategy that protects publicly released reference photos by disrupting model fine-tuning. While effective against unauthorized personalization, they do not address another practical setting in which personalization is authorized, but the resulting public outputs still leak identity information. To address this problem, we introduce a new defense setting, termed model-side output immunization, whose goal is to produce a personalized model that supports authorized personalization while reducing the identity linkability of public generations, with tunable control over the privacy-utility trade-off to accommodate diverse privacy needs. To this end, we propose Identity-Decoupled personalized Diffusion Models (IDDM), a model-side defense that integrates identity decoupling into the personalization pipeline. Concretely, IDDM follows an alternating procedure that interleaves short personalization updates with identity-decoupled data optimization, using a two-stage schedule to balance identity linkability suppression and generation utility. Extensive experiments across multiple datasets, diverse prompts, and state-of-the-art face recognition systems show that IDDM consistently reduces identity linkability while preserving high-quality personalized generation.

Related papers

• PIU: Proximity-guided Identity Unlearning in ID-Conditioned Diffusion Models [2.260041671242786]
  We study identity unlearning in Arc2Face, a state-of-the-art identity-conditioned latent diffusion model for face generation.<n>We introduce Proximity-guided Identity Unlearning (PIU), an anchor-guided framework for identity unlearning.
  arXiv  Detail & Related papers  (2026-05-21T10:55:56Z)
• Identity-Decoupled Anonymization for Visual Evidence in Multi-modal Retrieval-Augmented Generation [9.39195684989942]
  Identity-Decoupled MRAG is a framework that interposes a generative anonymization module between retrieval and generation.<n>Our approach consists of three components: (i)a disentangled variational encoder that factorizes each face into an identity code and a spatially-structured attribute code, regularized by a mutual-information penalty and a gradient-based independence term; (ii)a manifold-aware rejection sampler that replaces the identity code with a synthetic one guaranteed to be both distinct from the original and realistic; and (iii)a conditional latent diffusion generator that synthesizes the anonymized face from the replacement identity and the
  arXiv  Detail & Related papers  (2026-04-26T07:42:33Z)
• Reverse Personalization [48.09783075634403]
  We analyze the identity generation process and introduce a reverse personalization framework for face anonymization.<n>Unlike prior anonymization methods, which lack control over facial attributes, our framework supports attribute-controllable anonymization.
  arXiv  Detail & Related papers  (2025-12-28T16:06:55Z)
• Beyond Inference Intervention: Identity-Decoupled Diffusion for Face Anonymization [55.29071072675132]
  Face anonymization aims to conceal identity information while preserving non-identity attributes.<n>We propose textbfIDsuperscript2Face, a training-centric anonymization framework.<n>We show that IDtextsuperscript2Face outperforms existing methods in visual quality, identity suppression, and utility preservation.
  arXiv  Detail & Related papers  (2025-10-28T09:28:12Z)
• ID-Cloak: Crafting Identity-Specific Cloaks Against Personalized Text-to-Image Generation [54.14901999875917]
  We investigate the creation of identity-specific cloaks that safeguard all images belong to a specific identity.<n>We craft identity-specific cloaks with the proposed novel objective that encourages the cloak to guide the model away from its normal output.<n>Our method, along with the proposed identity-specific cloak setting, marks a notable advance in realistic privacy protection.
  arXiv  Detail & Related papers  (2025-02-12T03:52:36Z)
• Foundation Cures Personalization: Improving Personalized Models' Prompt Consistency via Hidden Foundation Knowledge [49.36669870661573]
  We propose FreeCure, a framework that improves the prompt consistency of personalization models.<n>We introduce a novel foundation-aware self-attention module, coupled with an inversion-based process to bring well-aligned attribute information to the personalization process.
  arXiv  Detail & Related papers  (2024-11-22T15:21:38Z)
• Infinite-ID: Identity-preserved Personalization via ID-semantics Decoupling Paradigm [31.06269858216316]
  We propose Infinite-ID, an ID-semantics decoupling paradigm for identity-preserved personalization. We introduce an identity-enhanced training, incorporating an additional image cross-attention module to capture sufficient ID information. We also introduce a feature interaction mechanism that combines a mixed attention module with an AdaIN-mean operation to seamlessly merge the two streams.
  arXiv  Detail & Related papers  (2024-03-18T13:39:53Z)
• Diff-Privacy: Diffusion-based Face Privacy Protection [58.1021066224765]
  In this paper, we propose a novel face privacy protection method based on diffusion models, dubbed Diff-Privacy. Specifically, we train our proposed multi-scale image inversion module (MSI) to obtain a set of SDM format conditional embeddings of the original image. Based on the conditional embeddings, we design corresponding embedding scheduling strategies and construct different energy functions during the denoising process to achieve anonymization and visual identity information hiding.
  arXiv  Detail & Related papers  (2023-09-11T09:26:07Z)
• Identity Encoder for Personalized Diffusion [57.1198884486401]
  We propose an encoder-based approach for personalization. We learn an identity encoder which can extract an identity representation from a set of reference images of a subject. We show that our approach consistently outperforms existing fine-tuning based approach in both image generation and reconstruction.
  arXiv  Detail & Related papers  (2023-04-14T23:32:24Z)
• Towards Privacy-Preserving Person Re-identification via Person Identify Shift [19.212691296927165]
  Person re-identification (ReID) requires preserving the privacy of pedestrian images used by ReID methods. We propose a novel de-identification method designed explicitly for person ReID, named Person Identify Shift (PIS) PIS shifts each pedestrian image from the current identity to another with a new identity, resulting in images still preserving the relative identities.
  arXiv  Detail & Related papers  (2022-07-15T06:58:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.