Vaccine: Perturbation-aware Alignment for Large Language Model
- URL: http://arxiv.org/abs/2402.01109v3
- Date: Thu, 29 Feb 2024 07:15:13 GMT
- Title: Vaccine: Perturbation-aware Alignment for Large Language Model
- Authors: Tiansheng Huang, Sihao Hu, Ling Liu
- Abstract summary: A few harmful data uploaded by users can easily trick the finetuning to produce an alignment-broken model.
We propose Vaccine, a perturbation-aware alignment technique to mitigate the security risk of users finetuning.
- Score: 8.601857354379096
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The new paradigm of finetuning-as-a-service introduces a new attack surface
for Large Language Models (LLMs): a few harmful data uploaded by users can
easily trick the finetuning to produce an alignment-broken model. We conduct an
empirical analysis and uncover a \textit{harmful embedding drift} phenomenon,
showing a probable cause of the alignment-broken effect. Inspired by our
findings, we propose Vaccine, a perturbation-aware alignment technique to
mitigate the security risk of users finetuning. The core idea of Vaccine is to
produce invariant hidden embeddings by progressively adding crafted
perturbation to them in the alignment phase. This enables the embeddings to
withstand harmful perturbation from un-sanitized user data in the finetuning
phase. Our results on open source mainstream LLMs (e.g., Llama2, Opt, Vicuna)
demonstrate that Vaccine can boost the robustness of alignment against harmful
prompts induced embedding drift while reserving reasoning ability towards
benign prompts. Our code is available at
\url{https://github.com/git-disl/Vaccine}.
Related papers
- Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation [86.05704141217036]
Black-box finetuning is an emerging interface for adapting state-of-the-art language models to user needs.
We introduce covert malicious finetuning, a method to compromise model safety via finetuning while evading detection.
arXiv Detail & Related papers (2024-06-28T17:05:46Z) - Breaking Free: How to Hack Safety Guardrails in Black-Box Diffusion Models! [52.0855711767075]
EvoSeed is an evolutionary strategy-based algorithmic framework for generating photo-realistic natural adversarial samples.
We employ CMA-ES to optimize the search for an initial seed vector, which, when processed by the Conditional Diffusion Model, results in the natural adversarial sample misclassified by the Model.
Experiments show that generated adversarial images are of high image quality, raising concerns about generating harmful content bypassing safety classifiers.
arXiv Detail & Related papers (2024-02-07T09:39:29Z) - Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injections [17.49244337226907]
We show that it is possible to conduct stealthy and persistent unalignment on large language models via backdoor injections.
Our proposed stealthy and persistent unalignment can successfully pass the safety evaluation while maintaining strong persistence against re-alignment defense.
arXiv Detail & Related papers (2023-11-15T23:52:05Z) - Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models [102.63973600144308]
Open-source large language models can be easily subverted to generate harmful content.
Experiments across 8 models released by 5 different organizations demonstrate the effectiveness of shadow alignment attack.
This study serves as a clarion call for a collective effort to overhaul and fortify the safety of open-source LLMs against malicious attackers.
arXiv Detail & Related papers (2023-10-04T16:39:31Z) - On the Exploitability of Instruction Tuning [103.8077787502381]
In this work, we investigate how an adversary can exploit instruction tuning to change a model's behavior.
We propose textitAutoPoison, an automated data poisoning pipeline.
Our results show that AutoPoison allows an adversary to change a model's behavior by poisoning only a small fraction of data.
arXiv Detail & Related papers (2023-06-28T17:54:04Z) - Are aligned neural networks adversarially aligned? [93.91072860401856]
adversarial users can construct inputs which circumvent attempts at alignment.
We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models.
We conjecture that improved NLP attacks may demonstrate this same level of adversarial control over text-only models.
arXiv Detail & Related papers (2023-06-26T17:18:44Z) - Exploring Model Dynamics for Accumulative Poisoning Discovery [62.08553134316483]
We propose a novel information measure, namely, Memorization Discrepancy, to explore the defense via the model-level information.
By implicitly transferring the changes in the data manipulation to that in the model outputs, Memorization Discrepancy can discover the imperceptible poison samples.
We thoroughly explore its properties and propose Discrepancy-aware Sample Correction (DSC) to defend against accumulative poisoning attacks.
arXiv Detail & Related papers (2023-06-06T14:45:24Z) - Denoising Diffusion Probabilistic Models as a Defense against
Adversarial Attacks [0.0]
This project evaluates the performance of Denoising Diffusion Probabilistic Models (DDPM) as a purification technique to defend against adversarial attacks.
We evaluate the approach on the PatchCamelyon data set for histopathologic scans of lymph node sections and find an improvement of the robust accuracy by up to 88% of the original model's accuracy.
arXiv Detail & Related papers (2023-01-17T13:27:53Z) - Disentangled Learning of Stance and Aspect Topics for Vaccine Attitude
Detection in Social Media [40.61499595293957]
We propose a novel semi-supervised approach for vaccine attitude detection, called VADet.
VADet is able to learn disentangled stance and aspect topics, and outperforms existing aspect-based sentiment analysis models on both stance detection and tweet clustering.
arXiv Detail & Related papers (2022-05-06T15:24:33Z) - Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability
of the Embedding Layers in NLP Models [27.100909068228813]
Recent studies have revealed a security threat to natural language processing (NLP) models, called the Backdoor Attack.
In this paper, we find that it is possible to hack the model in a data-free way by modifying one single word embedding vector.
Experimental results on sentiment analysis and sentence-pair classification tasks show that our method is more efficient and stealthier.
arXiv Detail & Related papers (2021-03-29T12:19:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.