The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks

• URL: http://arxiv.org/abs/2310.15469v2
• Date: Sun, 12 May 2024 07:52:22 GMT
• Title: The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks
• Authors: Xiaoyi Chen, Siyuan Tang, Rui Zhu, Shijun Yan, Lei Jin, Zihao Wang, Liya Su, Zhikun Zhang, XiaoFeng Wang, Haixu Tang,
• Abstract summary: We propose a novel attack, Janus, which exploits the fine-tuning interface to recover forgotten PIIs from the pre-training data in language models. Our experiment results show that Janus amplifies the privacy risks by over 10 times in comparison with the baseline. Our analysis validates that existing fine-tuning APIs provided by OpenAI and Azure AI Studio are susceptible to our Janus attack.
• Score: 19.364127374679253
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid advancements of large language models (LLMs) have raised public concerns about the privacy leakage of personally identifiable information (PII) within their extensive training datasets. Recent studies have demonstrated that an adversary could extract highly sensitive privacy data from the training data of LLMs with carefully designed prompts. However, these attacks suffer from the model's tendency to hallucinate and catastrophic forgetting (CF) in the pre-training stage, rendering the veracity of divulged PIIs negligible. In our research, we propose a novel attack, Janus, which exploits the fine-tuning interface to recover forgotten PIIs from the pre-training data in LLMs. We formalize the privacy leakage problem in LLMs and explain why forgotten PIIs can be recovered through empirical analysis on open-source language models. Based upon these insights, we evaluate the performance of Janus on both open-source language models and two latest LLMs, i.e., GPT-3.5-Turbo and LLaMA-2-7b. Our experiment results show that Janus amplifies the privacy risks by over 10 times in comparison with the baseline and significantly outperforms the state-of-the-art privacy extraction attacks including prefix attacks and in-context learning (ICL). Furthermore, our analysis validates that existing fine-tuning APIs provided by OpenAI and Azure AI Studio are susceptible to our Janus attack, allowing an adversary to conduct such an attack at a low cost.

Related papers

• Understanding Privacy Risks of Embeddings Induced by Large Language Models [75.96257812857554]
  Large language models show early signs of artificial general intelligence but struggle with hallucinations. One promising solution is to store external knowledge as embeddings, aiding LLMs in retrieval-augmented generation. Recent studies experimentally showed that the original text can be partially reconstructed from text embeddings by pre-trained language models.
  arXiv  Detail & Related papers  (2024-04-25T13:10:48Z)
• Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning [61.2224355547598]
  Open-sourcing of large language models (LLMs) accelerates application development, innovation, and scientific progress. Our investigation exposes a critical oversight in this belief. By deploying carefully designed demonstrations, our research demonstrates that base LLMs could effectively interpret and execute malicious instructions.
  arXiv  Detail & Related papers  (2024-04-16T13:22:54Z)
• Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models [112.48136829374741]
  In this paper, we unveil a new vulnerability: the privacy backdoor attack. When a victim fine-tunes a backdoored model, their training data will be leaked at a significantly higher rate than if they had fine-tuned a typical model. Our findings highlight a critical privacy concern within the machine learning community and call for a reevaluation of safety protocols in the use of open-source pre-trained models.
  arXiv  Detail & Related papers  (2024-04-01T16:50:54Z)
• Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs [61.04246774006429]
  We introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent. We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements. Our findings show that instruction-tuned models can expose pre-training data as much as their base-models, if not more so, and using instructions proposed by other LLMs can open a new avenue of automated attacks.
  arXiv  Detail & Related papers  (2024-03-05T19:32:01Z)
• The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) [56.67603627046346]
  Retrieval-augmented generation (RAG) is a powerful technique to facilitate language model with proprietary and private data. In this work, we conduct empirical studies with novel attack methods, which demonstrate the vulnerability of RAG systems on leaking the private retrieval database.
  arXiv  Detail & Related papers  (2024-02-23T18:35:15Z)
• Beyond Gradient and Priors in Privacy Attacks: Leveraging Pooler Layer Inputs of Language Models in Federated Learning [24.059033969435973]
  This paper presents a two-stage privacy attack strategy that targets the vulnerabilities in the architecture of contemporary language models. Our comparative experiments demonstrate superior attack performance across various datasets and scenarios. We call for the community to recognize and address these potential privacy risks in designing large language models.
  arXiv  Detail & Related papers  (2023-12-10T01:19:59Z)
• FLTrojan: Privacy Leakage Attacks against Federated Language Models Through Selective Weight Tampering [2.2194815687410627]
  We show how a malicious client can leak the privacy-sensitive data of some other users in FL even without any cooperation from the server. Our best-performing method improves the membership inference recall by 29% and achieves up to 71% private data reconstruction.
  arXiv  Detail & Related papers  (2023-10-24T19:50:01Z)
• Large Language Models Can Be Good Privacy Protection Learners [53.07930843882592]
  We introduce Privacy Protection Language Models (PPLM), a novel paradigm for fine-tuning language models. Our work offers a theoretical analysis for model design and delves into various techniques such as corpus curation, penalty-based unlikelihood in training loss, and instruction-based tuning. In particular, instruction tuning with both positive and negative examples, stands out as a promising method, effectively protecting private data while enhancing the model's knowledge.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey [43.063650238194384]
  Large Language Models (LLMs) have shown greatly enhanced performance in recent years, attributed to increased size and extensive training data. Training data memorization in Machine Learning models scales with model size, particularly concerning for LLMs. Memorized text sequences have the potential to be directly leaked from LLMs, posing a serious threat to data privacy.
  arXiv  Detail & Related papers  (2023-09-27T15:15:23Z)
• Knowledge Sanitization of Large Language Models [4.722882736419499]
  Large language models (LLMs) trained on a large corpus of Web data can potentially reveal sensitive or confidential information. Our technique efficiently fine-tunes these models using the Low-Rank Adaptation (LoRA) method. Experimental results in a closed-book question-answering task show that our straightforward method not only minimizes particular knowledge leakage but also preserves the overall performance of LLMs.
  arXiv  Detail & Related papers  (2023-09-21T07:49:55Z)
• Knowledge Unlearning for Mitigating Privacy Risks in Language Models [31.322818016245087]
  We propose knowledge unlearning as an alternative method to reduce privacy risks for language models. We show that simply applying the unlikelihood training objective to target token sequences is effective at forgetting them. We show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori.
  arXiv  Detail & Related papers  (2022-10-04T10:18:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.