Vision Language Model Helps Private Information De-Identification in Vision Data

• URL: http://arxiv.org/abs/2606.09132v1
• Date: Mon, 08 Jun 2026 07:30:20 GMT
• Title: Vision Language Model Helps Private Information De-Identification in Vision Data
• Abstract summary: We introduce VisShield, an end-to-end framework designed to enhance the privacy awareness of Visual Language Models (VLMs)<n>Our framework consists of two key components: a specialized instruction-tuning dataset OPTIC and a tailored training methodology.<n>Our approach ensures that VLMs recognize privacy-sensitive text and output precise bounding boxes for detected entities.
• Score: 55.425628316813174
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual Language Models (VLMs) have gained significant popularity due to their remarkable ability. While various methods exist to enhance privacy in text-based applications, privacy risks associated with visual inputs remain largely overlooked such as Protected Health Information (PHI) in medical images. To tackle this problem, two key tasks: accurately localizing sensitive text and processing it to ensure privacy protection should be performed. To address this issue, we introduce VisShield (Vision Privacy Shield), an end-to-end framework designed to enhance the privacy awareness of VLMs. Our framework consists of two key components: a specialized instruction-tuning dataset OPTIC (Optical Privacy Text Instruction Collection) and a tailored training methodology. The dataset provides diverse privacy-oriented prompts that guide VLMs to perform targeted Optical Character Recognition (OCR) for precise localization of sensitive text, while the training strategy ensures effective adaptation of VLMs to privacy-preserving tasks. Specifically, our approach ensures that VLMs recognize privacy-sensitive text and output precise bounding boxes for detected entities, allowing for effective masking of sensitive information. Extensive experiments demonstrate that our framework significantly outperforms existing approaches in handling private information, paving the way for privacy-preserving applications in vision-language models. Our dataset and code can be found here.

Related papers

• Assessing Privacy Preservation and Utility in Online Vision-Language Models [15.74713639305756]
  Images contain relationships that relate to Personally Identifiable Information (PII)<n>We investigate how the extraction of contextual relationships from images can lead to direct (explicit) or indirect (implicit) exposure of PII.<n>We propose methods to protect privacy while preserving the intended utility of the images in Vision Language Model (VLM)-based applications.
  arXiv  Detail & Related papers  (2026-04-06T20:44:46Z)
• Privacy-Preserving in Connected and Autonomous Vehicles Through Vision to Text Transformation [0.9831489366502302]
  This paper introduces a novel privacy-preserving framework that leverages feedback-based reinforcement learning (RL) and vision-language models (VLMs)<n>The main idea is to convert images into semantically equivalent textual descriptions, ensuring that scene-relevant information is retained while visual privacy is preserved.<n> Evaluation results demonstrate significant improvements in both privacy protection and textual quality.
  arXiv  Detail & Related papers  (2025-06-18T20:02:24Z)
• Activity Recognition on Avatar-Anonymized Datasets with Masked Differential Privacy [64.32494202656801]
  Privacy-preserving computer vision is an important emerging problem in machine learning and artificial intelligence.<n>We present anonymization pipeline that replaces sensitive human subjects in video datasets with synthetic avatars within context.<n>We also proposeMaskDP to protect non-anonymized but privacy sensitive background information.
  arXiv  Detail & Related papers  (2024-10-22T15:22:53Z)
• The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models [31.166994121531232]
  Vision-Language Models (VLMs) combine visual and textual understanding, rendering them well-suited for diverse tasks. These capabilities are built upon training on large amount of uncurated data crawled from the web. In this paper, we assess whether these vulnerabilities exist, focusing on identity leakage.
  arXiv  Detail & Related papers  (2024-08-02T12:36:13Z)
• Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning [62.224804688233]
  differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit. We study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.
  arXiv  Detail & Related papers  (2024-06-20T13:54:32Z)
• Little Data, Big Impact: Privacy-Aware Visual Language Models via Minimal Tuning [16.272314073324626]
  We evaluate tenofthe-art Visual As Language Models (VLMs) and identify limitations in their understanding of visual privacy.<n>To address this, we introduce two compact, high-quality benchmarks, PrivBench-H and PrivTune.<n>We obtain substantial gains on all benchmarks, surpassing GPT-4, while maintaining strong performance on other tasks.
  arXiv  Detail & Related papers  (2024-05-27T17:59:25Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• Diff-Privacy: Diffusion-based Face Privacy Protection [58.1021066224765]
  In this paper, we propose a novel face privacy protection method based on diffusion models, dubbed Diff-Privacy. Specifically, we train our proposed multi-scale image inversion module (MSI) to obtain a set of SDM format conditional embeddings of the original image. Based on the conditional embeddings, we design corresponding embedding scheduling strategies and construct different energy functions during the denoising process to achieve anonymization and visual identity information hiding.
  arXiv  Detail & Related papers  (2023-09-11T09:26:07Z)
• SPAct: Self-supervised Privacy Preservation for Action Recognition [73.79886509500409]
  Existing approaches for mitigating privacy leakage in action recognition require privacy labels along with the action labels from the video dataset. Recent developments of self-supervised learning (SSL) have unleashed the untapped potential of the unlabeled data. We present a novel training framework which removes privacy information from input video in a self-supervised manner without requiring privacy labels.
  arXiv  Detail & Related papers  (2022-03-29T02:56:40Z)
• Privacy-Adaptive BERT for Natural Language Understanding [20.821155542969947]
  We study how to improve the effectiveness of NLU models under a Local Privacy setting using BERT. We propose privacy-adaptive LM pretraining methods and demonstrate that they can significantly improve model performance on privatized text input.
  arXiv  Detail & Related papers  (2021-04-15T15:01:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.