Unsafe2Safe: Controllable Image Anonymization for Downstream Utility

• URL: http://arxiv.org/abs/2603.28605v1
• Date: Mon, 30 Mar 2026 15:54:47 GMT
• Title: Unsafe2Safe: Controllable Image Anonymization for Downstream Utility
• Abstract summary: We present Unsafe2Safe, a fully automated pipeline that detects privacy-prone images and rewrites only their sensitive regions.<n>We use a vision-language model to inspect images for privacy risks, generate paired private and public captions, and prompt a large language model to produce structured, identity-neutral edit instructions.<n>Stage 2 employs instruction-driven diffusion editors to apply these dual textual prompts, producing privacy-safe images.
• Score: 2.5484740244438524
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale image datasets frequently contain identifiable or sensitive content, raising privacy risks when training models that may memorize and leak such information. We present Unsafe2Safe, a fully automated pipeline that detects privacy-prone images and rewrites only their sensitive regions using multimodally guided diffusion editing. Unsafe2Safe operates in two stages. Stage 1 uses a vision-language model to (i) inspect images for privacy risks, (ii) generate paired private and public captions that respectively include and omit sensitive attributes, and (iii) prompt a large language model to produce structured, identity-neutral edit instructions conditioned on the public caption. Stage 2 employs instruction-driven diffusion editors to apply these dual textual prompts, producing privacy-safe images that preserve global structure and task-relevant semantics while neutralizing private content. To measure anonymization quality, we introduce a unified evaluation suite covering Quality, Cheating, Privacy, and Utility dimensions. Across MS-COCO, Caltech101, and MIT Indoor67, Unsafe2Safe reduces face similarity, text similarity, and demographic predictability by large margins, while maintaining downstream model accuracy comparable to training on raw data. Fine-tuning diffusion editors on our automatically generated triplets (private caption, public caption, edit instruction) further improves both privacy protection and semantic fidelity. Unsafe2Safe provides a scalable, principled solution for constructing large, privacy-safe datasets without sacrificing visual consistency or downstream utility.

Related papers

• DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models [47.98028812152569]
  We propose a privacy threat model named identity-affiliation learning.<n>An attacker fine-tunes a VLM using only a few private photos of a target individual.<n>This model enables unauthorized exposure of the target user's private information upon input of their photos.<n>To mitigate this privacy risk, we propose DP2-VL, the first dataset Protection framework for private photos.
  arXiv  Detail & Related papers  (2026-03-25T04:30:42Z)
• Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models [47.866853046761044]
  We find that diverse, subtle patterns in training data can degrade contextual privacy.<n>Fine-tuned models lose their ability to reason about contextual privacy norms.<n>Our results reveal a critical gap in current safety evaluations.
  arXiv  Detail & Related papers  (2026-01-21T17:53:06Z)
• SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Models [74.11062256255387]
  Text-to-image models are highly vulnerable to adversarial prompts, which can bypass safety measures and produce harmful content.<n>We introduce SafeGuider, a two-step framework designed for robust safety control without compromising generation quality.<n>SafeGuider demonstrates exceptional effectiveness in minimizing attack success rates, achieving a maximum rate of only 5.48% across various attack scenarios.
  arXiv  Detail & Related papers  (2025-10-05T10:24:48Z)
• T2UE: Generating Unlearnable Examples from Text Descriptions [60.111026156038264]
  Unlearnable Examples (UEs) have emerged as a promising countermeasure against unauthorized model training.<n>We introduce textbfText-to-Unlearnable Example (T2UE), a novel framework that enables users to generate UEs using only text descriptions.
  arXiv  Detail & Related papers  (2025-08-05T05:10:14Z)
• Privacy-Preserving in Connected and Autonomous Vehicles Through Vision to Text Transformation [0.9831489366502302]
  This paper introduces a novel privacy-preserving framework that leverages feedback-based reinforcement learning (RL) and vision-language models (VLMs)<n>The main idea is to convert images into semantically equivalent textual descriptions, ensuring that scene-relevant information is retained while visual privacy is preserved.<n> Evaluation results demonstrate significant improvements in both privacy protection and textual quality.
  arXiv  Detail & Related papers  (2025-06-18T20:02:24Z)
• Diff-Privacy: Diffusion-based Face Privacy Protection [58.1021066224765]
  In this paper, we propose a novel face privacy protection method based on diffusion models, dubbed Diff-Privacy. Specifically, we train our proposed multi-scale image inversion module (MSI) to obtain a set of SDM format conditional embeddings of the original image. Based on the conditional embeddings, we design corresponding embedding scheduling strategies and construct different energy functions during the denoising process to achieve anonymization and visual identity information hiding.
  arXiv  Detail & Related papers  (2023-09-11T09:26:07Z)
• PRO-Face S: Privacy-preserving Reversible Obfuscation of Face Images via Secure Flow [69.78820726573935]
  We name it PRO-Face S, short for Privacy-preserving Reversible Obfuscation of Face images via Secure flow-based model. In the framework, an Invertible Neural Network (INN) is utilized to process the input image along with its pre-obfuscated form, and generate the privacy protected image that visually approximates to the pre-obfuscated one.
  arXiv  Detail & Related papers  (2023-07-18T10:55:54Z)
• InfoScrub: Towards Attribute Privacy by Targeted Obfuscation [77.49428268918703]
  We study techniques that allow individuals to limit the private information leaked in visual data. We tackle this problem in a novel image obfuscation framework. We find our approach generates obfuscated images faithful to the original input images, and additionally increase uncertainty by 6.2$times$ (or up to 0.85 bits) over the non-obfuscated counterparts.
  arXiv  Detail & Related papers  (2020-05-20T19:48:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.