Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data

• URL: http://arxiv.org/abs/2409.11423v1
• Date: Thu, 12 Sep 2024 10:14:12 GMT
• Title: Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data
• Authors: Atilla Akkus, Mingjie Li, Junjie Chu, Michael Backes, Yang Zhang, Sinem Sav,
• Abstract summary: Large language models (LLMs) have shown considerable success in a range of domain-specific tasks, especially after fine-tuning. Fine-tuning with real-world data usually leads to privacy risks, particularly when the fine-tuning samples exist in the pre-training data. We present an empirical analysis of this underexplored issue by investigating a key question: "Does fine-tuning with LLM-generated data enhance privacy, or does it pose additional privacy risks?"
• Score: 19.93652217096443
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) have shown considerable success in a range of domain-specific tasks, especially after fine-tuning. However, fine-tuning with real-world data usually leads to privacy risks, particularly when the fine-tuning samples exist in the pre-training data. To avoid the shortcomings of real data, developers often employ methods to automatically generate synthetic data for fine-tuning, as data generated by traditional models are often far away from the real-world pertaining data. However, given the advanced capabilities of LLMs, the distinction between real data and LLM-generated data has become negligible, which may also lead to privacy risks like real data. In this paper, we present an empirical analysis of this underexplored issue by investigating a key question: "Does fine-tuning with LLM-generated data enhance privacy, or does it pose additional privacy risks?" Based on the structure of LLM's generated data, our research focuses on two primary approaches to fine-tuning with generated data: supervised fine-tuning with unstructured generated data and self-instruct tuning. The number of successful Personal Information Identifier (PII) extractions for Pythia after fine-tuning our generated data raised over $20\%$. Furthermore, the ROC-AUC score of membership inference attacks for Pythia-6.9b after self-instruct methods also achieves more than $40\%$ improvements on ROC-AUC score than base models. The results indicate the potential privacy risks in LLMs when fine-tuning with the generated data.

Related papers

• Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models [79.65071553905021]
  We propose Data Advisor, a method for generating data that takes into account the characteristics of the desired dataset. Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation.
  arXiv  Detail & Related papers  (2024-10-07T17:59:58Z)
• HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection [44.225151701532454]
  In this paper, we introduce a new framework HARMONIC for tabular data generation and evaluation. Our framework achieves equivalent performance to existing methods with better privacy, which also demonstrates our evaluation framework for the effectiveness of synthetic data and privacy risks.
  arXiv  Detail & Related papers  (2024-08-06T03:21:13Z)
• Robust Utility-Preserving Text Anonymization Based on Large Language Models [80.5266278002083]
  Text anonymization is crucial for sharing sensitive data while maintaining privacy. Existing techniques face the emerging challenges of re-identification attack ability of Large Language Models. This paper proposes a framework composed of three LLM-based components -- a privacy evaluator, a utility evaluator, and an optimization component.
  arXiv  Detail & Related papers  (2024-07-16T14:28:56Z)
• Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data [51.41288763521186]
  Retrieval-augmented generation (RAG) enhances the outputs of language models by integrating relevant information retrieved from external knowledge sources. RAG systems may face severe privacy risks when retrieving private data. We propose using synthetic data as a privacy-preserving alternative for the retrieval data.
  arXiv  Detail & Related papers  (2024-06-20T22:53:09Z)
• MALLM-GAN: Multi-Agent Large Language Model as Generative Adversarial Network for Synthesizing Tabular Data [10.217822818544475]
  We propose a framework to generate synthetic (tabular) data powered by large language models (LLMs) Our approach significantly enhance the quality of synthetic data generation in common scenarios with small sample sizes. Our results demonstrate that our model outperforms several state-of-art models regarding generating higher quality synthetic data for downstream tasks while keeping privacy of the real data.
  arXiv  Detail & Related papers  (2024-06-15T06:26:17Z)
• Aligning Large Language Models with Self-generated Preference Data [72.99676237703099]
  We propose a new framework that boosts the alignment of large language models (LLMs) with human preferences. Our key idea is leveraging the human prior knowledge within the small (seed) data. We introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data.
  arXiv  Detail & Related papers  (2024-06-06T18:01:02Z)
• FewFedPIT: Towards Privacy-preserving and Few-shot Federated Instruction Tuning [54.26614091429253]
  Federated instruction tuning (FedIT) is a promising solution, by consolidating collaborative training across multiple data owners. FedIT encounters limitations such as scarcity of instructional data and risk of exposure to training data extraction attacks. We propose FewFedPIT, designed to simultaneously enhance privacy protection and model performance of federated few-shot learning.
  arXiv  Detail & Related papers  (2024-03-10T08:41:22Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• PrivGen: Preserving Privacy of Sequences Through Data Generation [14.579475552088688]
  Sequential data can serve as a basis for research that will lead to improved processes. Access and use of such data is usually limited or not permitted at all due to concerns about violating user privacy. We propose PrivGen, an innovative method for generating data that maintains patterns and characteristics of the source data.
  arXiv  Detail & Related papers  (2020-02-23T05:43:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.