Quantifying Privacy Risks of Prompts in Visual Prompt Learning

• URL: http://arxiv.org/abs/2310.11970v1
• Date: Wed, 18 Oct 2023 13:51:27 GMT
• Title: Quantifying Privacy Risks of Prompts in Visual Prompt Learning
• Abstract summary: We perform the first comprehensive privacy assessment of prompts learned by visual prompt learning. Our empirical evaluation shows that the prompts are vulnerable to both attacks. We show that membership inference attacks against prompts can be successful with relaxed adversarial assumptions.
• Score: 33.244695390157595
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale pre-trained models are increasingly adapted to downstream tasks through a new paradigm called prompt learning. In contrast to fine-tuning, prompt learning does not update the pre-trained model's parameters. Instead, it only learns an input perturbation, namely prompt, to be added to the downstream task data for predictions. Given the fast development of prompt learning, a well-generalized prompt inevitably becomes a valuable asset as significant effort and proprietary data are used to create it. This naturally raises the question of whether a prompt may leak the proprietary information of its training data. In this paper, we perform the first comprehensive privacy assessment of prompts learned by visual prompt learning through the lens of property inference and membership inference attacks. Our empirical evaluation shows that the prompts are vulnerable to both attacks. We also demonstrate that the adversary can mount a successful property inference attack with limited cost. Moreover, we show that membership inference attacks against prompts can be successful with relaxed adversarial assumptions. We further make some initial investigations on the defenses and observe that our method can mitigate the membership inference attacks with a decent utility-defense trade-off but fails to defend against property inference attacks. We hope our results can shed light on the privacy risks of the popular prompt learning paradigm. To facilitate the research in this direction, we will share our code and models with the community.

Related papers

• Boundary-targeted Membership Inference Attacks on Safety Classifiers [70.20833439671131]
  Safety classifiers are trained on sensitive datasets including discussions of self-harm and mental health.<n>We introduce a new boundary-targeted selection strategy that identifies low confidence examples.<n>Experiments show that an adversary can recover 19% of the conversations a safety classifier flagged as indicating user distress, at a 5% false-positive rate.<n>This is $3.5$ times more than attacking using state-of-the-art MIA methods alone.
  arXiv  Detail & Related papers  (2026-05-21T12:05:22Z)
• Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning [24.914116986408327]
  We propose PromptMIA, a membership inference attack tailored to federated prompt-tuning.<n>We empirically show that PromptMIA consistently attains high advantage in this game across diverse benchmark datasets.<n>The results highlight non-trivial challenges for current defenses and offer insights into their limitations.
  arXiv  Detail & Related papers  (2026-01-10T17:50:05Z)
• Retrieval-augmented Prompt Learning for Pre-trained Foundation Models [101.13972024610733]
  We present RetroPrompt, which aims to achieve a balance between memorization and generalization.<n>Unlike traditional prompting methods, RetroPrompt incorporates a retrieval mechanism throughout the input, training, and inference stages.<n>We conduct comprehensive experiments on a variety of datasets across natural language processing and computer vision tasks to demonstrate the superior performance of our proposed approach.
  arXiv  Detail & Related papers  (2025-12-23T08:15:34Z)
• Revisiting Prompt Pretraining of Vision-Language Models [13.888505919946578]
  We propose a general framework termed Revisiting Prompt Pretraining (RPP) RPP targets at improving the fitting and generalization ability from two aspects: prompt structure and prompt supervision. We additionally utilize soft labels derived from zero-shot probability predictions provided by a pretrained Contrastive Language Image Pretraining (CLIP) teacher model.
  arXiv  Detail & Related papers  (2024-09-10T02:36:13Z)
• Detecting, Explaining, and Mitigating Memorization in Diffusion Models [49.438362005962375]
  We introduce a straightforward yet effective method for detecting memorized prompts by inspecting the magnitude of text-conditional predictions. Our proposed method seamlessly integrates without disrupting sampling algorithms, and delivers high accuracy even at the first generation step. Building on our detection strategy, we unveil an explainable approach that shows the contribution of individual words or tokens to memorization.
  arXiv  Detail & Related papers  (2024-07-31T16:13:29Z)
• Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning [49.242828934501986]
  Multimodal contrastive learning has emerged as a powerful paradigm for building high-quality features. backdoor attacks subtly embed malicious behaviors within the model during training. We introduce an innovative token-based localized forgetting training regime.
  arXiv  Detail & Related papers  (2024-03-24T18:33:15Z)
• On the Role of Attention in Prompt-tuning [90.97555030446563]
  We study prompt-tuning for one-layer attention architectures and study contextual mixture-models. We show that softmax-prompt-attention is provably more expressive than softmax-self-attention and linear-prompt-attention. We also provide experiments that verify our theoretical insights on real datasets and demonstrate how prompt-tuning enables the model to attend to context-relevant information.
  arXiv  Detail & Related papers  (2023-06-06T06:23:38Z)
• Students Parrot Their Teachers: Membership Inference on Model Distillation [54.392069096234074]
  We study the privacy provided by knowledge distillation to both the teacher and student training sets. Our attacks are strongest when student and teacher sets are similar, or when the attacker can poison the teacher set.
  arXiv  Detail & Related papers  (2023-03-06T19:16:23Z)
• Protecting Split Learning by Potential Energy Loss [70.81375125791979]
  We focus on the privacy leakage from the forward embeddings of split learning. We propose the potential energy loss to make the forward embeddings become more 'complicated'
  arXiv  Detail & Related papers  (2022-10-18T06:21:11Z)
• Exploring the Universal Vulnerability of Prompt-based Learning Paradigm [21.113683206722207]
  We find that prompt-based learning bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. We explore this universal vulnerability by either injecting backdoor triggers or searching for adversarial triggers on pre-trained language models using only plain text.
  arXiv  Detail & Related papers  (2022-04-11T16:34:10Z)
• Where Did You Learn That From? Surprising Effectiveness of Membership Inference Attacks Against Temporally Correlated Data in Deep Reinforcement Learning [114.9857000195174]
  A major challenge to widespread industrial adoption of deep reinforcement learning is the potential vulnerability to privacy breaches. We propose an adversarial attack framework tailored for testing the vulnerability of deep reinforcement learning algorithms to membership inference attacks.
  arXiv  Detail & Related papers  (2021-09-08T23:44:57Z)
• Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models [2.8021833233819486]
  Continual (or "incremental") learning approaches are employed when additional knowledge or tasks need to be learned from subsequent batches or from streaming data. We show that an intelligent adversary can take advantage of a continual learning algorithm's capabilities of retaining existing knowledge over time. We show that the adversary can create a "false memory" about any task by inserting carefully-designed backdoor samples to the test instances of that task.
  arXiv  Detail & Related papers  (2021-02-16T18:45:01Z)
• FaceLeaks: Inference Attacks against Transfer Learning Models via Black-box Queries [2.7564955518050693]
  We investigate if one can leak or infer private information without interacting with the teacher model directly. We propose novel strategies to infer from aggregate-level information. Our study indicates that information leakage is a real privacy threat to the transfer learning framework widely used in real-life situations.
  arXiv  Detail & Related papers  (2020-10-27T03:02:40Z)
• Sampling Attacks: Amplification of Membership Inference Attacks by Repeated Queries [74.59376038272661]
  We introduce sampling attack, a novel membership inference technique that unlike other standard membership adversaries is able to work under severe restriction of no access to scores of the victim model. We show that a victim model that only publishes the labels is still susceptible to sampling attacks and the adversary can recover up to 100% of its performance. For defense, we choose differential privacy in the form of gradient perturbation during the training of the victim model as well as output perturbation at prediction time.
  arXiv  Detail & Related papers  (2020-09-01T12:54:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.