Are You Copying My Prompt? Protecting the Copyright of Vision Prompt for VPaaS via Watermark

• URL: http://arxiv.org/abs/2405.15161v1
• Date: Fri, 24 May 2024 02:31:03 GMT
• Title: Are You Copying My Prompt? Protecting the Copyright of Vision Prompt for VPaaS via Watermark
• Abstract summary: Visual Prompt Learning (VPL) differs from traditional fine-tuning methods in reducing significant resource consumption by avoiding updating pre-trained model parameters. Developers of Visual Prompts as a Service (V) have emerged. These developers profit by providing well-crafted prompts to authorized customers. However, a significant drawback is that prompts can be easily copied and redistributed, threatening the intellectual property of V developers.
• Score: 4.071771108540602
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Prompt Learning (VPL) differs from traditional fine-tuning methods in reducing significant resource consumption by avoiding updating pre-trained model parameters. Instead, it focuses on learning an input perturbation, a visual prompt, added to downstream task data for making predictions. Since learning generalizable prompts requires expert design and creation, which is technically demanding and time-consuming in the optimization process, developers of Visual Prompts as a Service (VPaaS) have emerged. These developers profit by providing well-crafted prompts to authorized customers. However, a significant drawback is that prompts can be easily copied and redistributed, threatening the intellectual property of VPaaS developers. Hence, there is an urgent need for technology to protect the rights of VPaaS developers. To this end, we present a method named \textbf{WVPrompt} that employs visual prompt watermarking in a black-box way. WVPrompt consists of two parts: prompt watermarking and prompt verification. Specifically, it utilizes a poison-only backdoor attack method to embed a watermark into the prompt and then employs a hypothesis-testing approach for remote verification of prompt ownership. Extensive experiments have been conducted on three well-known benchmark datasets using three popular pre-trained models: RN50, BIT-M, and Instagram. The experimental results demonstrate that WVPrompt is efficient, harmless, and robust to various adversarial operations.

Related papers

• GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models [19.133064005999838]
  GhostPrompt is an adversarial prompt that is optimized once and reused to steer VLM outputs toward attacker-specified responses across diverse images.<n>Our method achieves an improvement of over 30% in attack success rates compared to state-of-the-art (SoTA) baselines, while reducing computation time by 70%.
  arXiv  Detail & Related papers  (2026-07-22T02:33:56Z)
• Rethinking Prompt Design for Inference-time Scaling in Text-to-Visual Generation [63.042451267669485]
  We propose Prompt Redesign for Inference-time Scaling, a framework that adaptively revises the prompt during inference in response to scaled visual generations.<n>We introduce a new verifier, element-level factual correction, which evaluates the alignment between prompt attributes and generated visuals at a fine-grained level.<n>Experiments on both text-to-image and text-to-video benchmarks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-12-03T07:54:05Z)
• The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation [40.73687553764341]
  Text-to-video (T2V) generative models, trained on large-scale datasets, are sensitive to input prompts. We introduce textbfRAPO, a novel textbfRetrieval-textbfAugmented textbfPrompt textbfOptimization framework.
  arXiv  Detail & Related papers  (2025-04-16T03:33:25Z)
• Revisiting Prompt Pretraining of Vision-Language Models [13.888505919946578]
  We propose a general framework termed Revisiting Prompt Pretraining (RPP) RPP targets at improving the fitting and generalization ability from two aspects: prompt structure and prompt supervision. We additionally utilize soft labels derived from zero-shot probability predictions provided by a pretrained Contrastive Language Image Pretraining (CLIP) teacher model.
  arXiv  Detail & Related papers  (2024-09-10T02:36:13Z)
• Why Are My Prompts Leaked? Unraveling Prompt Extraction Threats in Customized Large Language Models [15.764672596793352]
  We analyze the underlying mechanism of prompt leakage, which we refer to as prompt memorization, and develop corresponding defending strategies. We find that current LLMs, even those with safety alignments like GPT-4, are highly vulnerable to prompt extraction attacks.
  arXiv  Detail & Related papers  (2024-08-05T12:20:39Z)
• PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer [76.39111896665585]
  Incremental Learning (IL) aims to learn deep models on sequential tasks continually. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples.
  arXiv  Detail & Related papers  (2024-07-04T10:37:58Z)
• Instructing Prompt-to-Prompt Generation for Zero-Shot Learning [116.33775552866476]
  We propose a textbfPrompt-to-textbfPrompt generation methodology (textbfP2P) to distill instructive visual prompts for transferable knowledge discovery. The core of P2P is to mine semantic-related instruction from prompt-conditioned visual features and text instruction on modal-sharing semantic concepts.
  arXiv  Detail & Related papers  (2024-06-05T07:59:48Z)
• AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs [51.217126257318924]
  We present a novel method that uses another Large Language Models, called the AdvPrompter, to generate human-readable adversarial prompts in seconds. We train the AdvPrompter using a novel algorithm that does not require access to the gradients of the TargetLLM. The trained AdvPrompter generates suffixes that veil the input instruction without changing its meaning, such that the TargetLLM is lured to give a harmful response.
  arXiv  Detail & Related papers  (2024-04-21T22:18:13Z)
• Prompt Stealing Attacks Against Large Language Models [5.421974542780941]
  We propose a novel attack against large language models (LLMs) Our proposed prompt stealing attack aims to steal these well-designed prompts based on the generated answers. Our experimental results show the remarkable performance of our proposed attacks.
  arXiv  Detail & Related papers  (2024-02-20T12:25:26Z)
• Prompt Backdoors in Visual Prompt Learning [35.90535387625355]
  We propose BadVisualPrompt, a simple yet effective backdoor attack against VPL. We identify and address a new technical challenge related to interactions between the backdoor trigger and visual prompt.
  arXiv  Detail & Related papers  (2023-10-11T16:25:45Z)
• High-Fidelity Lake Extraction via Two-Stage Prompt Enhancement: Establishing a Novel Baseline and Benchmark [18.399869837003457]
  Lake extraction from remote sensing imagery is a complex challenge due to the varied lake shapes and data noise. This paper introduces a prompt-based dataset construction approach that provides approximate lake locations. We also propose a two-stage prompt enhancement framework, LEPrompter, with prompt-based and prompt-free stages during training.
  arXiv  Detail & Related papers  (2023-08-16T15:51:05Z)
• Effective Prompt Extraction from Language Models [70.00099540536382]
  We present a framework for measuring the effectiveness of prompt extraction attacks. In experiments with 3 different sources of prompts and 11 underlying large language models, we find that simple text-based attacks can in fact reveal prompts with high probability. Our framework determines with high precision whether an extracted prompt is the actual secret prompt, rather than a model hallucination.
  arXiv  Detail & Related papers  (2023-07-13T16:15:08Z)
• Rethinking Visual Prompt Learning as Masked Visual Token Modeling [106.71983630652323]
  We propose Visual Prompt learning as masked visual Token Modeling (VPTM) to transform the downstream visual classification into the pre-trained masked visual token prediction. VPTM is the first visual prompt method on the generative pre-trained visual model, which achieves consistency between pre-training and downstream visual classification by task reformulation.
  arXiv  Detail & Related papers  (2023-03-09T02:43:10Z)
• Optimizing Prompts for Text-to-Image Generation [97.61295501273288]
  Well-designed prompts can guide text-to-image models to generate amazing images. But the performant prompts are often model-specific and misaligned with user input. We propose prompt adaptation, a framework that automatically adapts original user input to model-preferred prompts.
  arXiv  Detail & Related papers  (2022-12-19T16:50:41Z)
• BadPrompt: Backdoor Attacks on Continuous Prompts [16.780478840918995]
  We conduct the first study on the vulnerability of the continuous prompt learning algorithm to backdoor attacks. We propose BadPrompt, a lightweight and task-adaptive algorithm, to backdoor attack continuous prompts. We evaluate the performance of BadPrompt on five datasets and two continuous prompt models.
  arXiv  Detail & Related papers  (2022-11-27T04:23:18Z)
• Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models [116.25562358482962]
  State-of-the-art neural language models can be used to solve ad-hoc language tasks without the need for supervised training. PromptIDE allows users to experiment with prompt variations, visualize prompt performance, and iteratively optimize prompts.
  arXiv  Detail & Related papers  (2022-08-16T17:17:53Z)
• Prompt-aligned Gradient for Prompt Tuning [63.346864107288766]
  We present Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from forgetting the general knowledge learned from vision-language models (VLMs) ProGrad only updates the prompt whose gradient is aligned to the "general direction", which is represented as the gradient of the KL loss of the pre-defined prompt prediction. Experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods.
  arXiv  Detail & Related papers  (2022-05-30T06:05:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.