DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models

• URL: http://arxiv.org/abs/2603.23925v1
• Date: Wed, 25 Mar 2026 04:30:42 GMT
• Title: DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models
• Abstract summary: We propose a privacy threat model named identity-affiliation learning.<n>An attacker fine-tunes a VLM using only a few private photos of a target individual.<n>This model enables unauthorized exposure of the target user's private information upon input of their photos.<n>To mitigate this privacy risk, we propose DP2-VL, the first dataset Protection framework for private photos.
• Score: 47.98028812152569
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in visual-language alignment have endowed vision-language models (VLMs) with fine-grained image understanding capabilities. However, this progress also introduces new privacy risks. This paper first proposes a novel privacy threat model named identity-affiliation learning: an attacker fine-tunes a VLM using only a few private photos of a target individual, thereby embedding associations between the target facial identity and their private property and social relationships into the model's internal representations. Once deployed via public APIs, this model enables unauthorized exposure of the target user's private information upon input of their photos. To benchmark VLMs' susceptibility to such identity-affiliation leakage, we introduce the first identity-affiliation dataset comprising seven typical scenarios appearing in private photos. Each scenario is instantiated with multiple identity-centered photo-description pairs. Experimental results demonstrate that mainstream VLMs like LLaVA, Qwen-VL, and MiniGPT-v2, can recognize facial identities and infer identity-affiliation relationships by fine-tuning on small-scale private photographic dataset, and even on synthetically generated datasets. To mitigate this privacy risk, we propose DP2-VL, the first Dataset Protection framework for private photos that leverages Data Poisoning. Though optimizing imperceptible perturbations by pushing the original representations toward an antithetical region, DP2-VL induces a dataset-level shift in the embedding space of VLMs'encoders. This shift separates protected images from clean inference images, causing fine-tuning on the protected set to overfit. Extensive experiments demonstrate that DP2-VL achieves strong generalization across models, robustness to diverse post-processing operations, and consistent effectiveness across varying protection ratios.

Related papers

• Unsafe2Safe: Controllable Image Anonymization for Downstream Utility [2.5484740244438524]
  We present Unsafe2Safe, a fully automated pipeline that detects privacy-prone images and rewrites only their sensitive regions.<n>We use a vision-language model to inspect images for privacy risks, generate paired private and public captions, and prompt a large language model to produce structured, identity-neutral edit instructions.<n>Stage 2 employs instruction-driven diffusion editors to apply these dual textual prompts, producing privacy-safe images.
  arXiv  Detail & Related papers  (2026-03-30T15:54:47Z)
• T2UE: Generating Unlearnable Examples from Text Descriptions [60.111026156038264]
  Unlearnable Examples (UEs) have emerged as a promising countermeasure against unauthorized model training.<n>We introduce textbfText-to-Unlearnable Example (T2UE), a novel framework that enables users to generate UEs using only text descriptions.
  arXiv  Detail & Related papers  (2025-08-05T05:10:14Z)
• VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models [15.158545794377169]
  We frame the preservation of privacy in Vision-Language Models as an adversarial attack problem.<n>We propose a novel attack strategy that selectively conceals information within designated Region Of Interests in an image.<n> Experimental results across three state-of-the-art VLMs demonstrate up to 98% reduction in detecting targeted ROIs.
  arXiv  Detail & Related papers  (2025-07-11T19:34:01Z)
• The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework [28.25933078258213]
  A new privacy risk is associated with the ability to infer sensitive attributes from personal images.<n>This threat is particularly severe given that modern apps can easily access users' photo albums.<n>In this work, we construct PAPI, the largest dataset for studying private attribute profiling in personal images.<n>We also propose HolmesEye, a hybrid agentic framework that combines VLMs and LLMs to enhance privacy inference.
  arXiv  Detail & Related papers  (2025-05-25T13:22:10Z)
• Private Attribute Inference from Images with Vision-Language Models [2.9373912230684565]
  Vision-language models (VLMs) are capable of understanding both images and text. We evaluate 7 state-of-the-art VLMs, finding that they can infer various personal attributes at up to 77.6% accuracy. We observe that accuracy scales with the general capabilities of the models, implying that future models can be misused as stronger inferential adversaries.
  arXiv  Detail & Related papers  (2024-04-16T14:42:49Z)
• Diff-Privacy: Diffusion-based Face Privacy Protection [58.1021066224765]
  In this paper, we propose a novel face privacy protection method based on diffusion models, dubbed Diff-Privacy. Specifically, we train our proposed multi-scale image inversion module (MSI) to obtain a set of SDM format conditional embeddings of the original image. Based on the conditional embeddings, we design corresponding embedding scheduling strategies and construct different energy functions during the denoising process to achieve anonymization and visual identity information hiding.
  arXiv  Detail & Related papers  (2023-09-11T09:26:07Z)
• Attribute-preserving Face Dataset Anonymization via Latent Code Optimization [64.4569739006591]
  We present a task-agnostic anonymization procedure that directly optimize the images' latent representation in the latent space of a pre-trained GAN. We demonstrate through a series of experiments that our method is capable of anonymizing the identity of the images whilst -- crucially -- better-preserving the facial attributes.
  arXiv  Detail & Related papers  (2023-03-20T17:34:05Z)
• ConfounderGAN: Protecting Image Data Privacy with Causal Confounder [85.6757153033139]
  We propose ConfounderGAN, a generative adversarial network (GAN) that can make personal image data unlearnable to protect the data privacy of its owners. Experiments are conducted in six image classification datasets, consisting of three natural object datasets and three medical datasets.
  arXiv  Detail & Related papers  (2022-12-04T08:49:14Z)
• InfoScrub: Towards Attribute Privacy by Targeted Obfuscation [77.49428268918703]
  We study techniques that allow individuals to limit the private information leaked in visual data. We tackle this problem in a novel image obfuscation framework. We find our approach generates obfuscated images faithful to the original input images, and additionally increase uncertainty by 6.2$times$ (or up to 0.85 bits) over the non-obfuscated counterparts.
  arXiv  Detail & Related papers  (2020-05-20T19:48:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.